2024-03-29T11:06:49.161 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:06:49.161 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687 --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/orig.config.yaml 2024-03-29T11:06:49.197 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:06:49.276 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622687' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCxyo7B7VLJ3qOY7qa7kNNFZrBfrbkytLD55taMv4gDfkl4/Rx1bzjo7AJ1HANhDGdmag+SwnOr//NmJEn9Lwg4= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6b5dmWfdaAVZg90zCJVBO0VxQSE9VQJvV2te0DX5qtMVBzk4CufFZ1bdai4eSTz3kGaPT4rw7BrhuC4dddMUk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:06:49.276 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:06:49.277 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:06:49.277 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:06:49.278 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:06:49.303 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:06:49.303 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:06:49.303 INFO:teuthology.packaging:ref: None 2024-03-29T11:06:49.303 INFO:teuthology.packaging:tag: None 2024-03-29T11:06:49.303 INFO:teuthology.packaging:branch: main 2024-03-29T11:06:49.303 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:49.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:06:49.549 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:06:49.551 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:06:49.559 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:06:49.559 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:06:49.579 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:06:49.592 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:06:49.603 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:06:49.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.435895', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCxyo7B7VLJ3qOY7qa7kNNFZrBfrbkytLD55taMv4gDfkl4/Rx1bzjo7AJ1HANhDGdmag+SwnOr//NmJEn9Lwg4='} 2024-03-29T11:06:49.647 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.437411', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6b5dmWfdaAVZg90zCJVBO0VxQSE9VQJvV2te0DX5qtMVBzk4CufFZ1bdai4eSTz3kGaPT4rw7BrhuC4dddMUk='} 2024-03-29T11:06:49.647 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:06:49.656 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:06:49.656 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:06:49.656 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:06:49.734 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7fdf8da0d0>, signals=[15]) 2024-03-29T11:06:49.735 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:06:49.743 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:06:49.743 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:06:49.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:49.852 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:06:49.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:49.945 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:06:49.954 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-03-29T11:06:49.971 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-03-29T11:06:49.972 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-29T11:06:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-29T11:06:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-03-29T11:06:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:50.030 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-29T11:06:50.053 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-29T11:06:50.070 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-29T11:06:50.070 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-29T11:06:50.126 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-03-29T11:06:50.126 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-03-29T11:06:50.126 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-03-29T11:06:50.126 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:50.127 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:50.127 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-29T11:06:50.145 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:06:50.154 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:06:50.161 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:06:50.161 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:50.163 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:50.182 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:06:50.190 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:06:50.190 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:50.220 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:50.253 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:06:50.308 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:06:50.325 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:06:50.325 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:06:50.325 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:50.326 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:50.342 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-29T11:06:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:50.343 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-29T11:06:50.397 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:50.398 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-361.el9.x86_64 2024-03-29T11:06:50.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-03-29T11:06:50.401 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:50.401 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-03-29T11:06:50.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-03-29T11:06:52.522 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:06:52.548 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:06:53.923 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:06:53.989 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:06:54.963 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:06:56.755 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-29T11:07:01.256 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.1 MB 00:00 2024-03-29T11:07:02.857 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-29T11:07:03.204 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 890 kB/s | 8.0 MB 00:09 2024-03-29T11:07:03.869 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-29T11:07:05.047 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-29T11:07:09.000 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:07:09.200 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:01 ago on Fri 29 Mar 2024 11:07:08 AM UTC. 2024-03-29T11:07:09.322 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-29T11:07:10.681 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-29T11:07:11.373 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:07:11.374 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:07:11.476 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:07:11.483 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:07:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:07:11.483 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:07:11.483 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-03-29T11:07:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:07:11.485 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 76 M 2024-03-29T11:07:11.485 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-03-29T11:07:11.485 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:07:11.651 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:07:12.102 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T11:07:12.727 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T11:07:13.044 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T11:07:13.051 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:13.052 INFO:teuthology.orchestra.run.smithi064.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-29T11:07:13.052 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:07:13.484 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:07:13.484 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:07:14.362 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:07:14.362 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:07:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:07:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:17.518 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:25.614 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:25.651 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:01.715 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:08:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:02.089 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:02.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:02.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:02.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:02.516 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:08:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:08:02.863 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:02.961 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:02.962 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:02.962 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:02.962 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:02.962 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:02.962 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:08:02.962 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:02.963 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:03.046 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:03.046 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:03.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:03.085 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:03.086 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-03-29T11:08:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:08:03.086 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:03.086 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:03.087 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:03.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-03-29T11:08:05.040 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:05.070 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-03-29T11:08:05.072 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-29T11:08:05.087 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.oK15RNCD6H 2024-03-29T11:08:05.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oK15RNCD6H 2024-03-29T11:08:05.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.oK15RNCD6H 2024-03-29T11:08:05.296 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.oK15RNCD6H is 6KB 2024-03-29T11:08:05.344 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.oK15RNCD6H 2024-03-29T11:08:05.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:05.432 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:05.432 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:05.432 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:05.432 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:05.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:05.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-03-29T11:08:35.579 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:08:35.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T11:08:35.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:53.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-29T11:09:02.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T11:09:02.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:05.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-29T11:09:17.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T11:09:17.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:49.831 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-29T11:09:50.216 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T11:09:50.216 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:09:50.216 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-29T11:09:50.229 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:09:50.229 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:09:50.229 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:09:50.230 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:09:51.230 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:09:51.230 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:09:51.329 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-03-29T11:09:51.372 DEBUG:teuthology.parallel:result is None 2024-03-29T11:13:47.058 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 54 kB/s | 21 MB 06:36 2024-03-29T11:13:52.025 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:13:54.247 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:13:54.248 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:13:54.346 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:13:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:13:54.352 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-03-29T11:13:54.352 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:13:54.352 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 76 M 2024-03-29T11:13:54.352 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 123 M 2024-03-29T11:13:54.352 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:13:54.534 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T11:13:55.510 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-03-29T11:13:55.967 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:13:56.401 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 16 MB/s | 34 MB 00:02 2024-03-29T11:13:56.404 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:56.404 INFO:teuthology.orchestra.run.smithi136.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-29T11:13:56.404 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:13:56.786 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:13:56.786 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:13:57.598 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:13:57.598 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:13:59.887 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:14:00.684 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:14:00.812 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:03.559 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:03.694 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:08.421 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:08.522 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:13.269 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:42.830 INFO:teuthology.orchestra.run.smithi136.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:14:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:14:42.875 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:43.310 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:43.311 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:14:43.311 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:43.311 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:14:43.727 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:14:44.086 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:14:44.181 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:14:44.181 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:14:44.181 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:14:44.182 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:14:44.182 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:14:44.182 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:14:44.182 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:14:44.182 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:14:44.262 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:14:44.262 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:14:44.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:14:44.296 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:14:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:14:44.298 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:14:44.298 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:14:44.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:14:44.868 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-03-29T11:14:46.352 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:14:46.385 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-03-29T11:14:46.388 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-29T11:14:46.403 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.sCejP7ikz7 2024-03-29T11:14:46.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sCejP7ikz7 2024-03-29T11:14:46.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.sCejP7ikz7 2024-03-29T11:14:46.610 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.sCejP7ikz7 is 6KB 2024-03-29T11:14:46.659 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.sCejP7ikz7 2024-03-29T11:14:46.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:14:46.738 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:14:46.738 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:14:46.738 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:14:46.738 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:14:46.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:14:46.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-03-29T11:15:16.891 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:15:16.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T11:15:16.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:16.894 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:16:25.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T11:16:25.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:29.293 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-29T11:16:29.700 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T11:16:29.700 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:16:29.700 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-29T11:16:29.714 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:16:29.714 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:16:29.714 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:16:29.715 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:16:30.715 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:16:30.716 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:16:30.782 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-29T11:16:30.822 DEBUG:teuthology.parallel:result is None 2024-03-29T11:16:30.822 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:16:30.830 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:16:30.830 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:16:30.833 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:16:30.849 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:16:30.855 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:16:30.865 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:16:30.865 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:16:30.891 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:16:30.939 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:16:30.946 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:16:30.946 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:16:30.957 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:16:30.992 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:31.002 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:31.005 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:31.017 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:31.018 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:16:31.025 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:16:31.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:16:31.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:16:31.074 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:16:31.083 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:16:31.083 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:16:31.097 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:16:31.131 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:31.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:31.244 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:31.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:31.369 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:16:31.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:16:31.436 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:31.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:31.525 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:31.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:31.649 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:16:31.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:16:31.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-03-29T11:16:31.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-29T11:16:31.770 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:16:31.806 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:16:32.058 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:16:32.069 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:16:32.069 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:16:32.079 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:16:32.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-03-29T11:16:32.160 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-03-29T11:16:32.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-03-29T11:16:32.208 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-03-29T11:16:32.209 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:16:32.209 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-03-29T11:16:32.232 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-03-29T11:16:32.232 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-03-29T11:16:32.274 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-03-29T11:16:32.274 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:16:32.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:16:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:16:32.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:16:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:16:32.346 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:16:32.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:16:32.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:16:32.422 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:16:32.432 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:16:32.437 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:16:32.572 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:16:32.587 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:16:32.588 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:20:45.505 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-03-29T11:20:45.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T11:20:45.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:20:45.599 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-29T11:20:45.681 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T11:20:45.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T11:20:45.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:20:45.780 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-29T11:20:45.868 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T11:20:45.869 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:20:46.208 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:20:46.208 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:20:46.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:20:46.211 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:20:46.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:20:46.243 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:20:46.244 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:20:46.260 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:20:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:20:46.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-03-29T11:20:46.292 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-03-29T11:20:46.302 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-03-29T11:20:46.306 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-03-29T11:20:46.318 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:20:46.323 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:20:46.335 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:20:46.340 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:20:46.389 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:20:46.392 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:20:46.393 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:20:46.393 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-29T11:20:46.393 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.393 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.393 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.393 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:20:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-29T11:20:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:20:46.396 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:20:46.405 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:20:46.405 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:20:46.405 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:20:46.406 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:20:46.411 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:20:46.411 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:20:46.412 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:20:46.412 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:20:46.414 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:20:46.414 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:20:46.560 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:20:46.560 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:20:46.565 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:20:46.565 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:20:46.675 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:20:46.675 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:20:46.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:20:46.704 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:20:46.704 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:20:46.730 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:20:46.730 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:20:46.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:20:46.759 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:20:46.759 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:20:46.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:20:46.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:20:46.852 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:20:46.906 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:20:46.918 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-03-29T11:20:46.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-03-29T11:20:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-03-29T11:20:46.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-03-29T11:20:47.177 INFO:teuthology.orchestra.run.smithi136.stdout:74 files removed 2024-03-29T11:20:47.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-03-29T11:20:47.230 INFO:teuthology.orchestra.run.smithi064.stdout:74 files removed 2024-03-29T11:20:47.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-03-29T11:20:47.896 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 210 kB/s | 82 kB 00:00 2024-03-29T11:20:48.035 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 174 kB/s | 82 kB 00:00 2024-03-29T11:20:48.255 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T11:20:48.410 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T11:20:48.614 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-29T11:20:48.786 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-29T11:20:49.816 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 5.7 MB/s | 6.7 MB 00:01 2024-03-29T11:20:49.847 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 6.5 MB/s | 6.7 MB 00:01 2024-03-29T11:20:51.289 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T11:20:51.383 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-29T11:20:52.223 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T11:20:53.196 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-29T11:20:54.718 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-29T11:20:57.638 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-03-29T11:21:00.060 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-29T11:21:01.392 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 99 kB/s | 16 kB 00:00 2024-03-29T11:21:01.789 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T11:21:03.257 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-29T11:21:04.451 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-03-29T11:21:06.541 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-03-29T11:21:09.384 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-29T11:21:11.743 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-03-29T11:21:11.992 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:21:12.012 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:21:12.013 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:21:12.014 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:Install 30 Packages 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:12.015 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 58 M 2024-03-29T11:21:12.016 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:21:13.460 INFO:teuthology.orchestra.run.smithi136.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.3 MB/s | 5.3 MB 00:01 2024-03-29T11:21:13.518 INFO:teuthology.orchestra.run.smithi136.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 433 kB/s | 25 kB 00:00 2024-03-29T11:21:13.635 INFO:teuthology.orchestra.run.smithi136.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T11:21:13.735 INFO:teuthology.orchestra.run.smithi136.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-29T11:21:14.286 INFO:teuthology.orchestra.run.smithi136.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.3 MB/s | 5.1 MB 00:00 2024-03-29T11:21:14.438 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:21:14.450 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:21:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:21:14.450 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:21:14.450 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:21:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:21:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:21:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-03-29T11:21:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:14.454 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 58 M 2024-03-29T11:21:14.454 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:21:14.502 INFO:teuthology.orchestra.run.smithi136.stdout:(6/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 8.7 MB/s | 20 MB 00:02 2024-03-29T11:21:14.528 INFO:teuthology.orchestra.run.smithi136.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 186 kB/s | 45 kB 00:00 2024-03-29T11:21:14.553 INFO:teuthology.orchestra.run.smithi136.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T11:21:14.586 INFO:teuthology.orchestra.run.smithi136.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T11:21:14.612 INFO:teuthology.orchestra.run.smithi136.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.3 MB/s | 320 kB 00:00 2024-03-29T11:21:14.645 INFO:teuthology.orchestra.run.smithi136.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:21:14.671 INFO:teuthology.orchestra.run.smithi136.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:21:14.863 INFO:teuthology.orchestra.run.smithi136.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 486 kB/s | 106 kB 00:00 2024-03-29T11:21:14.888 INFO:teuthology.orchestra.run.smithi136.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 734 kB/s | 159 kB 00:00 2024-03-29T11:21:14.921 INFO:teuthology.orchestra.run.smithi136.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 769 kB/s | 44 kB 00:00 2024-03-29T11:21:14.997 INFO:teuthology.orchestra.run.smithi136.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-29T11:21:15.022 INFO:teuthology.orchestra.run.smithi136.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-29T11:21:15.047 INFO:teuthology.orchestra.run.smithi136.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T11:21:15.072 INFO:teuthology.orchestra.run.smithi136.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:21:15.098 INFO:teuthology.orchestra.run.smithi136.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 694 kB/s | 35 kB 00:00 2024-03-29T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 787 kB/s | 46 kB 00:00 2024-03-29T11:21:15.307 INFO:teuthology.orchestra.run.smithi136.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-03-29T11:21:15.415 INFO:teuthology.orchestra.run.smithi136.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-29T11:21:15.465 INFO:teuthology.orchestra.run.smithi136.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-03-29T11:21:15.533 INFO:teuthology.orchestra.run.smithi136.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T11:21:15.558 INFO:teuthology.orchestra.run.smithi136.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-29T11:21:15.600 INFO:teuthology.orchestra.run.smithi136.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-29T11:21:15.667 INFO:teuthology.orchestra.run.smithi136.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-29T11:21:15.708 INFO:teuthology.orchestra.run.smithi136.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-29T11:21:16.492 INFO:teuthology.orchestra.run.smithi136.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.1 MB/s | 3.4 MB 00:00 2024-03-29T11:21:16.542 INFO:teuthology.orchestra.run.smithi136.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.8 MB/s | 3.1 MB 00:00 2024-03-29T11:21:16.577 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 5.3 MB/s | 9.7 MB 00:01 2024-03-29T11:21:16.639 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 431 kB/s | 25 kB 00:00 2024-03-29T11:21:16.761 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.8 MB/s | 742 kB 00:00 2024-03-29T11:21:16.845 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.9 MB/s | 498 kB 00:00 2024-03-29T11:21:17.029 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 5.3 MB 00:02 2024-03-29T11:21:17.246 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 7.9 MB/s | 20 MB 00:02 2024-03-29T11:21:17.271 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 186 kB/s | 45 kB 00:00 2024-03-29T11:21:17.313 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:21:17.380 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.5 MB/s | 5.1 MB 00:00 2024-03-29T11:21:17.405 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.2 MB/s | 162 kB 00:00 2024-03-29T11:21:17.438 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 320 kB 00:00 2024-03-29T11:21:17.464 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T11:21:17.522 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 848 kB/s | 99 kB 00:00 2024-03-29T11:21:17.623 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-29T11:21:17.648 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 454 kB/s | 44 kB 00:00 2024-03-29T11:21:17.665 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 917 kB/s | 159 kB 00:00 2024-03-29T11:21:17.757 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-29T11:21:17.782 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-29T11:21:17.799 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 308 kB/s | 41 kB 00:00 2024-03-29T11:21:17.824 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:21:17.841 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 842 kB/s | 35 kB 00:00 2024-03-29T11:21:17.858 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 797 kB/s | 46 kB 00:00 2024-03-29T11:21:17.977 INFO:teuthology.orchestra.run.smithi136.stdout:(32/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.7 MB/s | 9.7 MB 00:05 2024-03-29T11:21:17.981 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:17.981 INFO:teuthology.orchestra.run.smithi136.stdout:Total 9.7 MB/s | 58 MB 00:05 2024-03-29T11:21:17.981 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:21:18.020 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:21:18.020 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:21:18.116 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-03-29T11:21:18.292 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-03-29T11:21:18.401 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-03-29T11:21:18.426 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 514 kB/s | 309 kB 00:00 2024-03-29T11:21:18.493 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:21:18.493 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:21:18.534 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-29T11:21:18.710 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-29T11:21:18.735 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-29T11:21:18.837 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-03-29T11:21:19.085 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-03-29T11:21:19.827 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.0 MB/s | 3.4 MB 00:01 2024-03-29T11:21:19.831 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:19.832 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T11:21:19.832 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:21:19.875 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:21:19.875 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:21:19.943 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:21:20.020 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:21:20.055 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:21:20.083 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:21:20.111 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:20.128 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:20.363 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:21:20.363 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:21:20.495 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:20.551 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:20.893 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:21:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:21.146 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:21:21.166 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:21:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:21:21.354 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:21.376 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:21.440 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:21.475 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:21:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:21:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:21:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:21:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:21:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:21:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:21:21.880 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:21:21.965 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:21:22.005 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:21:22.025 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:21:22.047 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:22.064 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:21:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:22.254 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:21:22.270 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:22.294 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:21:22.618 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:21:22.675 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:21:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:22.777 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:22.848 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:22.915 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:21:22.915 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:21:22.943 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:22.945 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:21:22.989 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:23.018 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:21:23.071 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:21:23.140 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:23.162 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:23.226 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:23.261 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:21:23.373 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:21:23.404 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:21:23.444 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:21:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:21:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:21:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:21:23.871 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:21:23.901 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:23.984 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:24.015 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:21:24.058 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:21:24.541 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:21:24.570 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:24.642 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:24.712 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:21:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:21:25.090 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:25.125 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:25.733 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:25.770 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:25.864 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:25.864 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:21:25.864 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:21:25.864 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:25.921 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:27.548 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:27.589 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:27.671 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:27.671 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:21:27.671 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:21:27.671 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:35.464 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:35.464 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-03-29T11:21:35.464 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-03-29T11:21:35.465 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-03-29T11:21:35.465 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-03-29T11:21:35.465 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-03-29T11:21:35.465 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-03-29T11:21:35.465 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-03-29T11:21:35.465 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:36.333 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:21:36.358 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:21:36.358 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:21:36.358 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:21:36.358 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:21:36.358 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:21:36.358 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:36.360 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:21:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:21:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:37.670 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:21:37.671 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:21:37.672 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:21:37.673 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:21:37.696 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:37.696 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-03-29T11:21:37.696 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-03-29T11:21:37.697 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-03-29T11:21:37.697 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-03-29T11:21:37.697 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-03-29T11:21:37.697 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-03-29T11:21:37.697 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-03-29T11:21:37.697 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:38.633 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:21:38.658 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:21:38.658 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:21:38.658 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:21:38.658 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:21:38.658 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:21:38.658 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:38.660 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:21:38.736 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:21:38.736 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:38.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:38.818 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:38.818 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-03-29T11:21:38.818 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.818 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:21:38.819 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:21:38.820 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:21:38.821 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:21:38.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:38.821 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:21:39.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-03-29T11:21:39.632 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:21:39.786 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:21:39.787 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:21:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:21:39.787 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:21:39.787 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:21:39.788 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-29T11:21:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:39.789 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 48 M 2024-03-29T11:21:39.790 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 204 M 2024-03-29T11:21:39.790 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:21:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:21:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:21:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:21:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:21:40.061 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:21:40.103 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T11:21:40.136 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:21:40.270 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 445 kB/s | 163 kB 00:00 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.921 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:40.923 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:21:41.012 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 214 kB/s | 187 kB 00:00 2024-03-29T11:21:41.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-03-29T11:21:41.846 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:21:42.003 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:21:42.004 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:21:42.005 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 48 M 2024-03-29T11:21:42.006 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 204 M 2024-03-29T11:21:42.006 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:21:42.216 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:21:42.325 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 300 kB 00:00 2024-03-29T11:21:42.375 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:21:42.400 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 458 kB/s | 163 kB 00:00 2024-03-29T11:21:42.645 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-29T11:21:42.648 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:42.649 INFO:teuthology.orchestra.run.smithi136.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T11:21:42.663 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:21:42.685 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:21:42.686 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:21:42.925 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 340 kB/s | 187 kB 00:00 2024-03-29T11:21:43.066 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:21:43.067 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:21:43.638 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:21:43.713 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:21:43.749 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:21:43.789 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:21:43.829 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:21:43.857 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:21:48.340 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:21:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:21:50.746 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:21:50.746 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:21:50.746 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:21:50.746 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:21:50.746 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:21:50.746 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:21:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:21:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:51.142 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:21:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:21:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:21:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:21:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:21:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:21:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:51.144 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:21:51.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-03-29T11:21:52.066 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:21:52.229 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:21:52.232 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:21:52.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:21:52.234 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout:Install 36 Packages 2024-03-29T11:21:52.235 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:21:52.236 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 30 M 2024-03-29T11:21:52.236 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 107 M 2024-03-29T11:21:52.236 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:21:52.416 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 4.6 MB/s | 47 MB 00:10 2024-03-29T11:21:52.419 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:52.419 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.6 MB/s | 48 MB 00:10 2024-03-29T11:21:52.433 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:21:52.454 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:21:52.454 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:21:52.853 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:21:52.853 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:21:52.924 INFO:teuthology.orchestra.run.smithi136.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-29T11:21:53.366 INFO:teuthology.orchestra.run.smithi136.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-03-29T11:21:53.451 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:21:53.508 INFO:teuthology.orchestra.run.smithi136.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 2.1 MB 00:00 2024-03-29T11:21:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:21:53.575 INFO:teuthology.orchestra.run.smithi136.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T11:21:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:21:53.626 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:21:53.676 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:21:53.691 INFO:teuthology.orchestra.run.smithi136.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:21:53.706 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:21:53.733 INFO:teuthology.orchestra.run.smithi136.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:21:53.767 INFO:teuthology.orchestra.run.smithi136.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-29T11:21:53.800 INFO:teuthology.orchestra.run.smithi136.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:21:53.834 INFO:teuthology.orchestra.run.smithi136.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:21:53.876 INFO:teuthology.orchestra.run.smithi136.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:21:53.943 INFO:teuthology.orchestra.run.smithi136.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:21:53.968 INFO:teuthology.orchestra.run.smithi136.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 103 kB 00:00 2024-03-29T11:21:54.002 INFO:teuthology.orchestra.run.smithi136.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:21:54.036 INFO:teuthology.orchestra.run.smithi136.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:21:54.361 INFO:teuthology.orchestra.run.smithi136.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 384 kB/s | 125 kB 00:00 2024-03-29T11:21:54.420 INFO:teuthology.orchestra.run.smithi136.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 383 kB/s | 22 kB 00:00 2024-03-29T11:21:54.562 INFO:teuthology.orchestra.run.smithi136.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-29T11:21:54.729 INFO:teuthology.orchestra.run.smithi136.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 12 MB/s | 17 MB 00:01 2024-03-29T11:21:54.754 INFO:teuthology.orchestra.run.smithi136.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 226 kB/s | 43 kB 00:00 2024-03-29T11:21:54.846 INFO:teuthology.orchestra.run.smithi136.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:21:54.930 INFO:teuthology.orchestra.run.smithi136.stdout:(21/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-03-29T11:21:54.946 INFO:teuthology.orchestra.run.smithi136.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-29T11:21:54.972 INFO:teuthology.orchestra.run.smithi136.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2024-03-29T11:21:54.989 INFO:teuthology.orchestra.run.smithi136.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-03-29T11:21:55.014 INFO:teuthology.orchestra.run.smithi136.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 464 kB/s | 19 kB 00:00 2024-03-29T11:21:55.031 INFO:teuthology.orchestra.run.smithi136.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 275 kB/s | 23 kB 00:00 2024-03-29T11:21:55.056 INFO:teuthology.orchestra.run.smithi136.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 292 kB/s | 20 kB 00:00 2024-03-29T11:21:55.073 INFO:teuthology.orchestra.run.smithi136.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 788 kB/s | 46 kB 00:00 2024-03-29T11:21:55.099 INFO:teuthology.orchestra.run.smithi136.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T11:21:55.125 INFO:teuthology.orchestra.run.smithi136.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 324 kB/s | 16 kB 00:00 2024-03-29T11:21:55.149 INFO:teuthology.orchestra.run.smithi136.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T11:21:55.183 INFO:teuthology.orchestra.run.smithi136.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-03-29T11:21:55.217 INFO:teuthology.orchestra.run.smithi136.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-29T11:21:55.242 INFO:teuthology.orchestra.run.smithi136.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T11:21:55.267 INFO:teuthology.orchestra.run.smithi136.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T11:21:55.367 INFO:teuthology.orchestra.run.smithi136.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-03-29T11:21:55.371 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:55.371 INFO:teuthology.orchestra.run.smithi136.stdout:Total 9.5 MB/s | 30 MB 00:03 2024-03-29T11:21:55.396 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:21:55.439 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:21:55.439 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:21:55.852 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:21:55.852 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:21:57.205 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:21:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:21:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:21:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:21:57.418 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:21:57.529 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:21:57.562 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:21:57.653 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:21:57.723 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:21:57.765 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:21:57.836 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:21:57.890 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:21:57.929 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:21:57.978 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:21:58.034 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:21:58.220 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:21:58.266 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:21:58.297 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:21:58.331 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:21:58.362 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:21:58.413 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:21:58.437 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:21:58.472 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:21:58.488 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:21:58.505 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:21:58.545 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:21:58.580 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:21:58.607 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:21:58.695 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:21:58.730 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:21:58.801 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:21:58.937 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:21:59.064 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:21:59.231 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:21:59.260 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:21:59.285 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:21:59.286 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:21:59.286 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:21:59.286 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:21:59.286 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:21:59.286 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:00.728 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:00.754 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:00.754 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:00.754 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:22:00.754 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:00.754 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:00.754 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:22:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:22:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:22:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:22:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:22:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:22:01.131 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:22:01.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:01.132 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:01.203 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:01.226 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:01.226 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:22:01.226 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:01.226 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:01.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-03-29T11:22:01.438 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:01.460 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:01.460 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:01.460 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:22:01.460 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:01.460 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:01.460 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:01.527 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:02.008 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:51 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:02.167 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:22:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:22:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:22:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:02.173 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-03-29T11:22:02.174 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-03-29T11:22:02.174 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:03.062 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-29T11:22:03.270 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:03.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:22:03.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:22:03.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:22:03.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:22:03.272 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:22:03.273 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:22:03.870 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 1.5 MB 00:00 2024-03-29T11:22:03.920 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-03-29T11:22:03.996 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T11:22:04.104 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 4.5 MB/s | 4.7 MB 00:01 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:04.168 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:22:04.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:22:04.170 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:04.171 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T11:22:04.196 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-29T11:22:04.213 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-29T11:22:04.230 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:22:04.247 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:22:04.281 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:22:04.331 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T11:22:04.356 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-29T11:22:04.381 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:22:04.406 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:22:04.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-03-29T11:22:04.582 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-29T11:22:04.682 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 416 kB/s | 125 kB 00:00 2024-03-29T11:22:04.841 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 586 kB/s | 151 kB 00:00 2024-03-29T11:22:04.908 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 193 kB/s | 43 kB 00:00 2024-03-29T11:22:04.975 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T11:22:05.022 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:05.050 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-29T11:22:05.120 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:05.125 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-29T11:22:05.150 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-29T11:22:05.173 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:05.174 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:05.174 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:05.184 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-03-29T11:22:05.209 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-03-29T11:22:05.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-03-29T11:22:05.254 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 281 kB/s | 20 kB 00:00 2024-03-29T11:22:05.270 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 761 kB/s | 46 kB 00:00 2024-03-29T11:22:05.321 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-29T11:22:05.363 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-29T11:22:05.388 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-29T11:22:05.463 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 897 kB/s | 90 kB 00:00 2024-03-29T11:22:05.480 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-03-29T11:22:05.547 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T11:22:05.598 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T11:22:05.623 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-29T11:22:05.818 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:05.968 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:05.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:05.970 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:05.970 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:05.970 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 747 k 2024-03-29T11:22:05.970 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 754 k 2024-03-29T11:22:05.970 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:06.190 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 7.2 MB/s | 17 MB 00:02 2024-03-29T11:22:06.194 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:06.194 INFO:teuthology.orchestra.run.smithi064.stdout:Total 7.4 MB/s | 30 MB 00:04 2024-03-29T11:22:06.219 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:06.265 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:06.266 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:06.551 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.3 MB/s | 747 kB 00:00 2024-03-29T11:22:06.552 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:06.552 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.3 MB/s | 747 kB 00:00 2024-03-29T11:22:06.552 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:06.557 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:06.557 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:06.563 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:06.563 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:06.685 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:06.685 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:07.110 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:07.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:07.591 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:07.873 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:07.873 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:07.873 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:07.873 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:07.873 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:07.874 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:08.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:22:08.126 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:22:08.169 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:22:08.203 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:22:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:22:08.327 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:22:08.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:22:08.464 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:22:08.539 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:22:08.613 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:22:08.654 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:22:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:22:08.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:22:08.807 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:08.808 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:08.809 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:08.809 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:08.809 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 145 k 2024-03-29T11:22:08.809 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 436 k 2024-03-29T11:22:08.809 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:08.827 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:22:08.890 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:22:09.082 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:22:09.131 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:22:09.166 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 407 kB/s | 145 kB 00:00 2024-03-29T11:22:09.166 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:09.166 INFO:teuthology.orchestra.run.smithi136.stdout:Total 405 kB/s | 145 kB 00:00 2024-03-29T11:22:09.166 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:09.169 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:22:09.174 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:09.174 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:09.212 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:22:09.225 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:09.226 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:22:09.338 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:22:09.362 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:09.378 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:09.411 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:22:09.434 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:09.452 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:22:09.457 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:09.457 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:09.458 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:22:09.458 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:09.495 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:22:09.530 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:22:09.623 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:22:09.660 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:22:09.735 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:22:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:22:10.000 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:10.167 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:22:10.190 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:22:10.215 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:22:10.215 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:10.215 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:22:10.215 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:22:10.215 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:22:10.216 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:10.454 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:10.454 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:10.454 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:10.454 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:10.455 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:10.455 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:10.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-03-29T11:22:11.312 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:11.411 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:11.463 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:11.465 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:11.465 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:11.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:11.759 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:11.759 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:11.759 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:22:11.759 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:11.759 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:11.759 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:12.098 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:12.248 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:12.248 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:22:12.248 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:12.248 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:12.252 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:12.253 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-29T11:22:12.253 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:12.253 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-29T11:22:12.254 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:12.254 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:22:12.254 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:22:12.254 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:22:12.254 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:22:12.254 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:22:12.255 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:22:12.255 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:22:12.255 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:22:12.255 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:22:12.255 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:22:12.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:22:12.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:22:12.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:22:12.256 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:12.256 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:12.256 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-29T11:22:12.257 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-29T11:22:12.257 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:12.257 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.9 M 2024-03-29T11:22:12.257 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-29T11:22:12.257 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:12.490 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:12.490 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:22:12.490 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:12.490 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:12.804 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 78 kB/s | 15 kB 00:00 2024-03-29T11:22:12.829 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 141 kB/s | 31 kB 00:00 2024-03-29T11:22:12.980 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T11:22:13.013 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T11:22:13.072 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.8 MB/s | 1.8 MB 00:00 2024-03-29T11:22:13.189 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-29T11:22:13.297 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-29T11:22:13.364 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-29T11:22:13.389 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-03-29T11:22:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.2 MB/s | 2.0 MB 00:00 2024-03-29T11:22:13.498 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T11:22:13.523 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 194 kB/s | 31 kB 00:00 2024-03-29T11:22:13.527 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:13.527 INFO:teuthology.orchestra.run.smithi136.stdout:Total 5.4 MB/s | 6.9 MB 00:01 2024-03-29T11:22:13.545 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:13.562 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:13.562 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:13.729 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:13.729 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:22:14.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:14.238 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:22:14.239 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:22:14.239 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:22:14.239 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:22:14.240 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:22:14.241 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:22:14.246 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:14.332 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:22:14.376 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:22:14.423 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:22:14.453 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:22:14.557 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:22:14.678 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:22:14.781 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:22:14.833 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:22:14.862 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:14.894 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:22:15.135 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:22:15.177 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:15.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:22:15.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:22:15.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:15.356 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:15.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-03-29T11:22:15.997 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:15.997 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:22:15.997 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:22:15.997 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:22:15.997 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:22:15.997 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:22:15.998 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:22:15.998 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:22:15.998 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:22:15.998 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:15.998 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:22:15.998 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:22:16.263 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:16.345 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:22:16.345 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:16.345 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:22:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:16.347 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:16.365 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:16.420 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:16.421 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:22:16.421 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:16.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-03-29T11:22:16.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:22:17.061 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:17.155 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:17.216 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:17.216 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:17.216 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 747 k 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 754 k 2024-03-29T11:22:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:17.309 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:17.310 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-29T11:22:17.310 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:17.310 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-29T11:22:17.310 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:22:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 37 M 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-03-29T11:22:17.312 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:17.623 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 258 kB/s | 32 kB 00:00 2024-03-29T11:22:17.648 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 691 kB/s | 17 kB 00:00 2024-03-29T11:22:17.690 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 830 kB/s | 34 kB 00:00 2024-03-29T11:22:17.893 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.1 MB/s | 747 kB 00:00 2024-03-29T11:22:17.894 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:17.894 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.1 MB/s | 747 kB 00:00 2024-03-29T11:22:17.894 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:17.899 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-29T11:22:17.900 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:17.900 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:17.906 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:17.906 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:17.986 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:18.090 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-29T11:22:18.132 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-29T11:22:18.308 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): ceph-mgr-diskprediction-local-19.0.0-14 9.1 MB/s | 7.4 MB 00:00 2024-03-29T11:22:18.359 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.5 MB/s | 3.0 MB 00:00 2024-03-29T11:22:18.456 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:18.608 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-29T11:22:18.750 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:22:18.776 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 471 kB/s | 195 kB 00:00 2024-03-29T11:22:18.913 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:18.959 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-29T11:22:18.961 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:18.961 INFO:teuthology.orchestra.run.smithi136.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T11:22:19.001 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:19.026 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:19.026 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:19.203 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:19.204 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:19.385 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:19.385 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:19.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:22:19.988 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:20.114 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:20.144 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:20.146 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:22:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:20.146 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-03-29T11:22:20.146 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 436 k 2024-03-29T11:22:20.146 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:20.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:22:20.238 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:22:20.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:22:20.375 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:22:20.533 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 376 kB/s | 145 kB 00:00 2024-03-29T11:22:20.533 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:20.533 INFO:teuthology.orchestra.run.smithi064.stdout:Total 375 kB/s | 145 kB 00:00 2024-03-29T11:22:20.534 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:20.542 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:20.542 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:20.551 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:22:20.582 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:22:20.597 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:20.597 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:20.667 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:22:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:20.816 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:20.816 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:22:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:21.319 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:22:21.435 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:22:21.828 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:21.828 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:21.828 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:21.828 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:21.828 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:21.829 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:22.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-03-29T11:22:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:22:22.676 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:22.778 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:22.833 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:22.835 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:22:22.835 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:22.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:22:23.479 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:23.639 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-29T11:22:23.640 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-29T11:22:23.641 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:23.641 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.9 M 2024-03-29T11:22:23.641 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-29T11:22:23.641 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:23.701 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:22:23.734 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:22:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 50 kB/s | 15 kB 00:00 2024-03-29T11:22:24.428 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 86 kB/s | 31 kB 00:00 2024-03-29T11:22:24.529 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:22:24.554 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-29T11:22:24.596 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.3 MB/s | 1.8 MB 00:00 2024-03-29T11:22:24.779 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-03-29T11:22:24.838 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-29T11:22:24.922 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-29T11:22:24.989 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 460 kB/s | 31 kB 00:00 2024-03-29T11:22:25.064 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-29T11:22:25.089 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 576 kB/s | 144 kB 00:00 2024-03-29T11:22:25.181 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.1 MB/s | 2.0 MB 00:00 2024-03-29T11:22:25.184 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:25.184 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.4 MB/s | 6.9 MB 00:01 2024-03-29T11:22:25.202 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:25.219 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:25.220 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:25.396 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:25.397 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:22:26.026 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:22:26.073 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:22:26.111 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:22:26.217 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:22:26.249 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:22:26.250 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:22:26.250 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:22:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:22:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:22:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:22:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:26.559 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:22:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:22:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:22:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:22:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:26.666 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:26.813 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:22:26.842 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:27.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:22:27.584 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:22:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:22:27.642 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:22:27.642 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:27.642 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:22:27.642 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:22:27.737 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:22:27.738 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-03-29T11:22:27.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:27.740 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.0 M 2024-03-29T11:22:27.740 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-29T11:22:27.740 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:27.927 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-03-29T11:22:27.952 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:28.011 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:28.012 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:28.017 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-29T11:22:28.045 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T11:22:28.069 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.0 MB/s | 211 kB 00:00 2024-03-29T11:22:28.094 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-29T11:22:28.120 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 160 kB/s | 52 kB 00:00 2024-03-29T11:22:28.162 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-03-29T11:22:28.187 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-03-29T11:22:28.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:22:28.237 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-29T11:22:28.263 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-29T11:22:28.296 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-29T11:22:28.346 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-03-29T11:22:28.348 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:28.348 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-03-29T11:22:28.349 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:28.361 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:28.361 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:28.452 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:28.804 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:28.964 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:28.965 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:22:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-29T11:22:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:28.968 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 37 M 2024-03-29T11:22:28.968 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-03-29T11:22:28.968 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:22:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:22:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:22:29.257 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:22:29.296 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 32 kB 00:00 2024-03-29T11:22:29.299 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:22:29.330 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 514 kB/s | 17 kB 00:00 2024-03-29T11:22:29.371 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-29T11:22:29.401 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:22:29.509 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:22:29.539 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T11:22:29.558 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:22:29.664 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2024-03-29T11:22:29.731 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-29T11:22:29.756 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:22:29.857 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 8.1 MB/s | 803 kB 00:00 2024-03-29T11:22:29.940 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 195 kB 00:00 2024-03-29T11:22:30.090 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T11:22:30.240 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 822 kB/s | 245 kB 00:00 2024-03-29T11:22:30.274 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:22:30.341 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:22:30.385 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:22:30.432 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:22:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:22:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:22:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:22:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:22:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:22:31.710 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:22:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:22:31.712 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:22:31.712 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:22:31.712 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:22:31.712 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:31.712 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:31.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:22:32.057 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.5 MB/s | 7.4 MB 00:02 2024-03-29T11:22:32.060 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:32.061 INFO:teuthology.orchestra.run.smithi064.stdout:Total 12 MB/s | 37 MB 00:03 2024-03-29T11:22:32.105 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:32.131 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:32.131 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:32.502 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:32.502 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:32.527 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:32.680 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:32.681 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:32.682 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T11:22:32.682 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-29T11:22:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:32.682 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.8 M 2024-03-29T11:22:32.682 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-03-29T11:22:32.682 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:33.235 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:22:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:22:33.410 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T11:22:33.436 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:22:33.473 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:22:33.485 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:22:33.610 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 344 kB/s | 138 kB 00:00 2024-03-29T11:22:33.665 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:22:33.689 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:22:33.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:22:33.794 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 202 kB/s | 58 kB 00:00 2024-03-29T11:22:33.894 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-03-29T11:22:33.919 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:22:33.923 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:33.923 INFO:teuthology.orchestra.run.smithi136.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-29T11:22:33.936 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:33.947 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:33.947 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:34.035 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:34.035 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:22:34.503 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:22:34.554 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:22:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:22:35.078 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:22:35.158 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:22:35.218 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:22:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:22:35.895 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:22:36.077 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:22:36.077 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:22:36.078 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:22:36.078 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:22:36.078 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:22:36.078 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:22:36.366 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:22:36.366 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:36.367 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:36.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-03-29T11:22:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:22:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:22:37.184 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:37.335 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:37.335 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:37.335 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-29T11:22:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:37.337 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 964 k 2024-03-29T11:22:37.337 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.8 M 2024-03-29T11:22:37.337 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:37.621 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 276 kB/s | 79 kB 00:00 2024-03-29T11:22:37.705 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 885 kB 00:00 2024-03-29T11:22:37.706 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:37.706 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.5 MB/s | 964 kB 00:00 2024-03-29T11:22:37.706 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:37.727 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:37.727 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:37.800 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:37.800 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:38.132 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:22:38.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:22:38.897 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:22:38.897 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:22:39.182 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:22:39.183 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:39.183 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:39.183 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:22:39.183 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:39.183 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:39.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-volume 2024-03-29T11:22:39.550 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:22:39.551 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:22:40.024 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:40.025 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:40.057 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:40.216 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:40.216 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T11:22:40.216 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-29T11:22:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:40.218 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 332 k 2024-03-29T11:22:40.218 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.5 M 2024-03-29T11:22:40.218 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:40.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:22:40.486 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 985 kB/s | 264 kB 00:00 2024-03-29T11:22:40.511 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 234 kB/s | 69 kB 00:00 2024-03-29T11:22:40.512 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:40.512 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T11:22:40.513 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:40.521 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:40.521 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:40.562 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:40.562 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:40.886 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:22:40.922 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:22:40.938 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:22:40.939 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:40.939 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:22:40.939 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:40.977 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:41.133 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:22:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-03-29T11:22:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:41.137 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.0 M 2024-03-29T11:22:41.137 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-29T11:22:41.137 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:41.640 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 122 kB/s | 24 kB 00:00 2024-03-29T11:22:41.657 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 83 kB/s | 18 kB 00:00 2024-03-29T11:22:41.674 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 223 kB/s | 52 kB 00:00 2024-03-29T11:22:41.717 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:22:41.824 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 950 kB/s | 174 kB 00:00 2024-03-29T11:22:41.949 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 453 kB/s | 132 kB 00:00 2024-03-29T11:22:41.975 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 700 kB/s | 211 kB 00:00 2024-03-29T11:22:41.992 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 256 kB/s | 43 kB 00:00 2024-03-29T11:22:42.029 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:22:42.029 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:42.029 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:42.029 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:42.029 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:22:42.030 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:42.030 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:42.109 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T11:22:42.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-03-29T11:22:42.542 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 2.4 MB/s | 1.0 MB 00:00 2024-03-29T11:22:42.651 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2024-03-29T11:22:42.743 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 977 kB/s | 90 kB 00:00 2024-03-29T11:22:42.929 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:43.080 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:43.080 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:43.081 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:43.082 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-03-29T11:22:43.082 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-03-29T11:22:43.082 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:43.210 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-cachetools-4.2.4-1.el9.noarch. 26 kB/s | 32 kB 00:01 2024-03-29T11:22:43.293 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 601 kB/s | 127 kB 00:00 2024-03-29T11:22:43.293 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:43.293 INFO:teuthology.orchestra.run.smithi136.stdout:Total 597 kB/s | 127 kB 00:00 2024-03-29T11:22:43.294 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:43.299 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:43.299 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:43.355 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:43.355 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:43.506 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:43.616 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:44.381 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:44.381 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:44.381 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:44.381 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:44.381 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:44.381 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:44.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-03-29T11:22:45.203 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:45.304 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:45.318 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 61 kB/s | 203 kB 00:03 2024-03-29T11:22:45.320 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:45.320 INFO:teuthology.orchestra.run.smithi064.stdout:Total 502 kB/s | 2.0 MB 00:04 2024-03-29T11:22:45.321 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:45.336 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:45.337 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:45.357 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:45.358 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:45.358 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:45.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-03-29T11:22:45.439 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:45.439 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:45.973 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:46.005 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:22:46.158 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:46.159 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 31 k 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 140 k 2024-03-29T11:22:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:46.212 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:22:46.268 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:22:46.304 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:22:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:22:46.405 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-29T11:22:46.405 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:46.405 INFO:teuthology.orchestra.run.smithi136.stdout:Total 127 kB/s | 31 kB 00:00 2024-03-29T11:22:46.406 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:46.409 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:46.409 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:46.433 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:46.433 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:46.450 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:22:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:22:46.568 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:22:46.669 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:47.081 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:47.354 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:47.354 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:22:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:22:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:22:47.493 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:22:47.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-03-29T11:22:47.522 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:22:48.072 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:48.170 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:48.223 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:48.224 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:48.224 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:22:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:22:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:22:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:22:48.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-03-29T11:22:48.611 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:22:48.611 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:48.611 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:48.611 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:48.611 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:48.612 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:48.839 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:48.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:22:48.938 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:48.988 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:48.989 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:48.990 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:49.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-03-29T11:22:49.455 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:49.613 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:49.614 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:49.614 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T11:22:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:49.614 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T11:22:49.614 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-29T11:22:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:49.616 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.8 M 2024-03-29T11:22:49.616 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-03-29T11:22:49.616 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:49.713 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:49.767 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:49.768 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:49.768 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:49.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-03-29T11:22:50.382 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:50.480 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:50.532 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:50.533 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:50.534 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:50.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-03-29T11:22:50.628 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-03-29T11:22:50.954 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 301 kB/s | 138 kB 00:00 2024-03-29T11:22:51.155 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:51.162 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.7 MB/s | 5.8 MB 00:00 2024-03-29T11:22:51.254 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:51.305 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:51.306 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:51.306 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:51.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-03-29T11:22:51.412 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 126 kB/s | 58 kB 00:00 2024-03-29T11:22:51.621 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 111 kB/s | 51 kB 00:00 2024-03-29T11:22:51.782 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 477 kB/s | 548 kB 00:01 2024-03-29T11:22:51.782 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:51.782 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.1 MB/s | 6.8 MB 00:02 2024-03-29T11:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:51.808 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:51.808 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:51.905 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:51.905 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:51.932 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:52.031 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:52.083 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:52.084 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T11:22:52.084 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:52.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-03-29T11:22:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:22:52.432 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:22:52.695 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:52.843 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:52.845 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:52.845 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-29T11:22:52.845 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:52.845 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:52.845 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 85 k 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 231 k 2024-03-29T11:22:52.846 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:52.991 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:22:53.050 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 417 kB/s | 85 kB 00:00 2024-03-29T11:22:53.050 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:53.050 INFO:teuthology.orchestra.run.smithi136.stdout:Total 414 kB/s | 85 kB 00:00 2024-03-29T11:22:53.051 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:53.057 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:53.057 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:53.077 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:22:53.112 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:53.113 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:53.139 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:22:53.168 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:22:53.259 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:53.763 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:53.919 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:22:53.919 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:22:53.919 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:22:53.919 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:22:53.919 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:22:53.920 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:22:54.023 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:54.023 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:54.023 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:54.023 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:54.023 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:54.023 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:54.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:22:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:22:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:54.303 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:54.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-03-29T11:22:54.847 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:55.011 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:55.012 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:55.012 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T11:22:55.012 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:55.012 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:55.012 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:22:55.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.1 M 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 13 M 2024-03-29T11:22:55.013 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:55.125 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:55.281 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:55.283 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-29T11:22:55.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:55.283 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 964 k 2024-03-29T11:22:55.283 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-03-29T11:22:55.283 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:55.378 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 8.5 MB/s | 3.1 MB 00:00 2024-03-29T11:22:55.379 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:55.380 INFO:teuthology.orchestra.run.smithi136.stdout:Total 8.4 MB/s | 3.1 MB 00:00 2024-03-29T11:22:55.380 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:55.389 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:55.389 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:55.414 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 601 kB/s | 79 kB 00:00 2024-03-29T11:22:55.465 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:55.465 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 885 kB 00:00 2024-03-29T11:22:55.882 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:55.883 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.6 MB/s | 964 kB 00:00 2024-03-29T11:22:55.883 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:55.905 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:55.905 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:55.912 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:55.971 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:55.982 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:55.982 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:55.997 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:55.997 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:55.997 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:22:55.998 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:22:55.998 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:22:55.998 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:56.177 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:22:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:22:57.097 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:22:57.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:22:57.230 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:57.230 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:57.230 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:57.230 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:57.230 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:57.231 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:22:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:57.435 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:22:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:22:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:57.435 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:22:57.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-03-29T11:22:57.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-03-29T11:22:58.080 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:21:09 AM UTC. 2024-03-29T11:22:58.229 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T11:22:58.230 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:58.230 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-29T11:22:58.230 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 171 k 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 494 k 2024-03-29T11:22:58.231 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T11:22:58.270 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:22:58.428 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:22:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 332 k 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-03-29T11:22:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:22:58.475 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 703 kB/s | 171 kB 00:00 2024-03-29T11:22:58.476 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:58.476 INFO:teuthology.orchestra.run.smithi136.stdout:Total 699 kB/s | 171 kB 00:00 2024-03-29T11:22:58.476 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T11:22:58.484 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T11:22:58.484 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T11:22:58.540 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T11:22:58.540 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T11:22:58.553 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 564 kB/s | 69 kB 00:00 2024-03-29T11:22:58.685 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T11:22:58.703 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 969 kB/s | 264 kB 00:00 2024-03-29T11:22:58.704 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:58.704 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:22:58.704 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:22:58.713 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:22:58.714 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:22:58.753 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:22:58.754 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:22:58.782 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:58.909 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:22:59.084 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:22:59.114 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:22:59.130 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:22:59.130 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:59.130 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:22:59.130 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:22:59.192 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:22:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:59.472 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T11:22:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:22:59.472 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T11:22:59.745 DEBUG:teuthology.parallel:result is None 2024-03-29T11:22:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:00.250 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:00.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-03-29T11:23:01.137 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:01.293 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:01.294 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T11:23:01.294 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:01.294 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:23:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-03-29T11:23:01.295 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:23:01.526 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 549 kB/s | 127 kB 00:00 2024-03-29T11:23:01.527 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:01.527 INFO:teuthology.orchestra.run.smithi064.stdout:Total 546 kB/s | 127 kB 00:00 2024-03-29T11:23:01.527 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:23:01.532 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:23:01.533 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:23:01.590 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:23:01.591 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:23:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:23:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:02.591 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:23:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:02.592 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:02.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-03-29T11:23:03.405 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:03.506 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:03.563 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:03.564 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:03.564 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:03.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-03-29T11:23:04.217 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:04.376 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:04.377 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T11:23:04.377 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:04.377 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:23:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:23:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 k 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 140 k 2024-03-29T11:23:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:23:04.671 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 106 kB/s | 31 kB 00:00 2024-03-29T11:23:04.672 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:04.672 INFO:teuthology.orchestra.run.smithi064.stdout:Total 106 kB/s | 31 kB 00:00 2024-03-29T11:23:04.672 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:23:04.675 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:23:04.675 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:23:04.707 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:23:04.708 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:23:04.830 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:23:04.946 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:05.616 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:23:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:05.616 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:05.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-03-29T11:23:06.343 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:06.444 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:06.498 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:06.500 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:06.500 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:06.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-03-29T11:23:07.130 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:07.231 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:07.287 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:07.288 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:07.288 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:07.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-03-29T11:23:07.940 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:08.042 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:08.097 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:08.098 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:08.099 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:08.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-03-29T11:23:08.736 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:08.837 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:08.892 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:08.893 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:08.893 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:08.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-03-29T11:23:09.521 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:09.622 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:09.677 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:09.678 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:09.679 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:09.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-03-29T11:23:10.307 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:10.409 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:10.464 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:10.465 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T11:23:10.465 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:10.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-03-29T11:23:11.087 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:11.242 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:11.242 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-03-29T11:23:11.243 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:23:11.568 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 263 kB/s | 85 kB 00:00 2024-03-29T11:23:11.569 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:11.569 INFO:teuthology.orchestra.run.smithi064.stdout:Total 261 kB/s | 85 kB 00:00 2024-03-29T11:23:11.569 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:23:11.577 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:23:11.577 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:23:11.635 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:23:11.635 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:23:11.787 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:23:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:12.574 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:23:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:12.574 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:12.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-03-29T11:23:13.373 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:13.532 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:23:13.533 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:13.534 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:23:13.534 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:13.534 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-03-29T11:23:13.534 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-03-29T11:23:13.534 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:23:14.022 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.3 MB/s | 3.1 MB 00:00 2024-03-29T11:23:14.022 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:14.023 INFO:teuthology.orchestra.run.smithi064.stdout:Total 6.3 MB/s | 3.1 MB 00:00 2024-03-29T11:23:14.023 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:23:14.033 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:23:14.033 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:23:14.110 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:23:14.110 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:23:14.563 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:23:14.631 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:14.655 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:14.655 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:23:14.655 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:23:14.655 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:23:14.655 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:23:14.655 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:15.783 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:23:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:15.783 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:16.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-03-29T11:23:16.592 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:21:11 AM UTC. 2024-03-29T11:23:16.747 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T11:23:16.747 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 171 k 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-03-29T11:23:16.748 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T11:23:17.101 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 487 kB/s | 171 kB 00:00 2024-03-29T11:23:17.101 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:17.101 INFO:teuthology.orchestra.run.smithi064.stdout:Total 485 kB/s | 171 kB 00:00 2024-03-29T11:23:17.102 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T11:23:17.110 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T11:23:17.110 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T11:23:17.168 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T11:23:17.168 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T11:23:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T11:23:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:17.851 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:18.193 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:18.193 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:18.193 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T11:23:18.193 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:18.193 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:18.193 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T11:23:18.445 DEBUG:teuthology.parallel:result is None 2024-03-29T11:23:18.445 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:23:18.588 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:23:18.609 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:18.609 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:18.610 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:23:18.611 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:23:18.903 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:23:18.928 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:18.928 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:18.928 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:23:18.930 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:23:18.930 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:18.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:23:18.958 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:18.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:23:18.999 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:23:19.000 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:19.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:23:19.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:23:19.094 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:19.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:23:19.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:23:19.194 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:23:19.194 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:19.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:23:19.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:23:19.287 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:19.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:23:19.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:23:19.385 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:23:19.385 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:19.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:23:19.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:23:19.479 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:19.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:23:19.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:23:19.575 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:23:19.649 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:23:19.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-03-29T11:23:19.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-03-29T11:23:19.679 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:23:19.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:23:19.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:23:19.746 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:23:19.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:23:19.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:23:19.823 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:23:19.823 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:23:19.824 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622687', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCxyo7B7VLJ3qOY7qa7kNNFZrBfrbkytLD55taMv4gDfkl4/Rx1bzjo7AJ1HANhDGdmag+SwnOr//NmJEn9Lwg4=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6b5dmWfdaAVZg90zCJVBO0VxQSE9VQJvV2te0DX5qtMVBzk4CufFZ1bdai4eSTz3kGaPT4rw7BrhuC4dddMUk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:23:19.824 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:23:19.852 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:23:19.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:23:19.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:23:19.935 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:19.935 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:23:19.986 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:23:19.986 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-03-29T11:23:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:23:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 11:23:14.962217886 +0000 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 11:20:23.564528967 +0000 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 11:20:23.564528967 +0000 2024-03-29T11:23:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 11:20:23.564528967 +0000 2024-03-29T11:23:20.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:23:20.105 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T11:23:20.105 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T11:23:20.105 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000253339 s, 2.0 MB/s 2024-03-29T11:23:20.106 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:23:20.161 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-03-29T11:23:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:23:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:20.216 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:23:20.217 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:20.217 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:20.217 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 11:23:14.962217886 +0000 2024-03-29T11:23:20.217 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 11:20:24.022519810 +0000 2024-03-29T11:23:20.217 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 11:20:24.022519810 +0000 2024-03-29T11:23:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 11:20:24.022519810 +0000 2024-03-29T11:23:20.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:23:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T11:23:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T11:23:20.282 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.00018208 s, 2.8 MB/s 2024-03-29T11:23:20.283 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:23:20.339 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 11:23:14.962217886 +0000 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 11:20:24.369512873 +0000 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 11:20:24.369512873 +0000 2024-03-29T11:23:20.394 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 11:20:24.369512873 +0000 2024-03-29T11:23:20.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:23:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T11:23:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T11:23:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.00022049 s, 2.3 MB/s 2024-03-29T11:23:20.461 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:23:20.517 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-03-29T11:23:20.570 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:23:20.570 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 11:23:14.963217866 +0000 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 11:20:24.704506175 +0000 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 11:20:24.704506175 +0000 2024-03-29T11:23:20.571 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 11:20:24.704506175 +0000 2024-03-29T11:23:20.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:23:20.636 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T11:23:20.636 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T11:23:20.636 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.00023779 s, 2.2 MB/s 2024-03-29T11:23:20.637 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:23:20.692 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:20.692 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:20.692 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:23:20.709 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:23:20.709 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 11:22:56.271225260 +0000 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 11:20:22.712074365 +0000 2024-03-29T11:23:20.766 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 11:20:22.712074365 +0000 2024-03-29T11:23:20.767 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 11:20:22.712074365 +0000 2024-03-29T11:23:20.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:23:20.835 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T11:23:20.835 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T11:23:20.835 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000218518 s, 2.3 MB/s 2024-03-29T11:23:20.837 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:23:20.894 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-29T11:23:20.951 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:23:20.951 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:20.951 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:23:20.951 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:20.952 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:20.952 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 11:22:56.271225260 +0000 2024-03-29T11:23:20.952 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 11:20:23.046067244 +0000 2024-03-29T11:23:20.952 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 11:20:23.046067244 +0000 2024-03-29T11:23:20.952 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 11:20:23.046067244 +0000 2024-03-29T11:23:20.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:23:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T11:23:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T11:23:21.020 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000218576 s, 2.3 MB/s 2024-03-29T11:23:21.020 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:23:21.077 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 11:22:56.272225239 +0000 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 11:20:23.374060252 +0000 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 11:20:23.374060252 +0000 2024-03-29T11:23:21.133 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 11:20:23.374060252 +0000 2024-03-29T11:23:21.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:23:21.200 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T11:23:21.200 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T11:23:21.200 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000197953 s, 2.6 MB/s 2024-03-29T11:23:21.201 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:23:21.257 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-29T11:23:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:23:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:21.312 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:23:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 11:22:56.272225239 +0000 2024-03-29T11:23:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 11:20:23.736052534 +0000 2024-03-29T11:23:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 11:20:23.736052534 +0000 2024-03-29T11:23:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 11:20:23.736052534 +0000 2024-03-29T11:23:21.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:23:21.378 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T11:23:21.378 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T11:23:21.378 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000235426 s, 2.2 MB/s 2024-03-29T11:23:21.379 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:23:21.435 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:21.435 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:23:21.435 INFO:tasks.ceph:Generating config... 2024-03-29T11:23:21.436 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:23:21.436 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:23:21.436 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:23:21.436 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:23:21.436 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:23:21.436 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:23:21.436 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:23:21.436 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:23:21.437 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:23:21.437 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:23:21.437 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:23:21.437 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:23:21.437 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:23:21.437 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:23:21.438 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:23:21.439 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:23:21.439 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:23:21.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:23:21.489 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:23:21.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:23:21.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:23:21.562 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.64:6789]'), ('mon.c', '[v1:172.21.15.64:6790]'), ('mon.b', '[v1:172.21.15.136:6789]')] 2024-03-29T11:23:21.563 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.136:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:23:21.563 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:21.563 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:23:21.617 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.64:6789]' --addv c '[v1:172.21.15.64:6790]' --addv b '[v1:172.21.15.136:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:21.702 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-03-29T11:23:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2024-03-29T11:23:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:23:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:23:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:23:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:23:21.704 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:23:21.718 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4e4fa36b-6805-4310-ad24-7c1ad3ae3959... 2024-03-29T11:23:21.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:23:21.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.807 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.808 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.809 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.136:6789]" 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: lockdep = True 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 1 2024-03-29T11:23:21.810 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr2 = False 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.811 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:23:21.812 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = zstd 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-03-29T11:23:21.813 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd class default list = * 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd class load list = * 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd client watch timeout = 120 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-03-29T11:23:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-03-29T11:23:21.815 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.816 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-03-29T11:23:21.817 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout: debug objecter = 20 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout: debug rados = 20 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-03-29T11:23:21.818 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-03-29T11:23:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:23:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-03-29T11:23:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.136:6789]" 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: lockdep = True 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: mon election default strategy = 1 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: ms bind msgr2 = False 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 5000 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: ms type = async 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-03-29T11:23:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression algorithm = zstd 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression mode = aggressive 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore zero block detection = True 2024-03-29T11:23:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 20 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 20 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 20 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 10 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd class default list = * 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd class load list = * 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd client watch timeout = 120 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify cached snaps = True 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify missing on start = True 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:23:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock override recovery settings = True 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue = debug_random 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue cut off = debug_random 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: osd shutdown pgref assert = True 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:23:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon min osdmap epochs = 50 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune interval = 2 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune min = 15 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub interval = 300 2024-03-29T11:23:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on pool no app = False 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: paxos service trim min = 10 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 20 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: debug objecter = 20 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: debug rados = 20 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2024-03-29T11:23:21.829 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2024-03-29T11:23:21.831 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:23:21.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:23:21.886 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:23:21.886 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:21.886 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:23:21.900 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:21.900 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:23:21.955 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:23:21.955 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:21.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:23:21.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:23:22.033 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:22.033 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:22.088 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:23:22.088 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:22.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:23:22.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:23:22.127 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:22.127 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:22.182 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:23:22.182 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:23:22.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:23:22.237 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:23:22.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:23:22.302 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:23:22.305 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:23:22.305 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:23:22.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:23:22.347 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:23:22.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:23:22.404 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:23:22.419 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:23:22.420 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:23:22.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:22.447 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:22.447 INFO:tasks.ceph:role: osd.0 2024-03-29T11:23:22.447 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:23:22.447 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:23:22.520 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:22.521 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:23.035 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T11:23:23.037 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T11:23:23.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:23.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:23.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:23.278 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:23.279 INFO:tasks.ceph:role: osd.1 2024-03-29T11:23:23.279 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:23:23.279 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:23.353 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:23.354 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:23.800 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T11:23:23.802 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T11:23:23.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:23.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:23.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:23.990 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:23.990 INFO:tasks.ceph:role: osd.2 2024-03-29T11:23:23.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:23:23.991 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:23:24.064 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:24.064 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:24.064 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:24.064 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:24.065 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:24.065 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:24.065 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:24.065 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:24.065 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:24.065 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:24.548 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T11:23:24.550 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T11:23:24.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:24.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:24.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:24.800 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:24.800 INFO:tasks.ceph:role: osd.3 2024-03-29T11:23:24.800 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:23:24.800 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:23:24.874 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:24.874 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:24.874 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:24.875 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:25.355 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T11:23:25.357 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T11:23:25.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:25.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:25.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:25.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:25.648+0000 7fea04e20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:23:25.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:25.648+0000 7fea04e20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:23:25.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:25.648+0000 7fea04e20640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:23:25.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:25.648+0000 7fea04e20640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:23:25.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:25.648+0000 7fea04e20640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:23:25.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:25.649+0000 7fea04e20640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:23:28.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:28.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:28.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:28.618+0000 7fd34fc1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:23:28.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:28.619+0000 7fd34fc1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:23:28.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:28.619+0000 7fd34fc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:23:28.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:28.619+0000 7fd34fc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:23:28.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:28.619+0000 7fd34fc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:23:28.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:28.619+0000 7fd34fc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:23:31.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:31.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:31.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:31.593+0000 7f9a12506640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:23:31.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:31.593+0000 7f9a12506640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:23:31.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:31.593+0000 7f9a12506640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:23:31.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:31.593+0000 7f9a12506640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:23:31.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:31.593+0000 7f9a12506640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:23:31.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:31.594+0000 7f9a12506640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:23:34.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:34.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:34.568+0000 7f5d398bf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:23:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:34.568+0000 7f5d398bf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:23:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:34.568+0000 7f5d398bf640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:23:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:34.568+0000 7f5d398bf640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:23:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:34.568+0000 7f5d398bf640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:23:34.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:34.568+0000 7f5d398bf640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:23:37.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:37.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:37.465 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:37.465 INFO:tasks.ceph:role: osd.4 2024-03-29T11:23:37.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:23:37.466 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:23:37.543 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:37.543 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:37.543 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:37.544 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:38.055 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T11:23:38.057 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T11:23:38.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:38.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:38.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:23:38.256 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:38.256 INFO:tasks.ceph:role: osd.5 2024-03-29T11:23:38.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:23:38.257 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:38.332 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:38.818 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T11:23:38.820 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T11:23:38.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:23:38.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:23:38.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:23:39.030 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:39.030 INFO:tasks.ceph:role: osd.6 2024-03-29T11:23:39.030 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:23:39.030 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:23:39.104 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:39.104 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:39.105 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:39.639 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T11:23:39.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T11:23:39.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:23:39.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:23:39.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:23:39.847 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:39.847 INFO:tasks.ceph:role: osd.7 2024-03-29T11:23:39.847 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:23:39.847 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:23:39.922 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:39.923 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:40.442 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T11:23:40.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T11:23:40.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:23:40.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:23:40.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:40.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:40.683+0000 7f05cca10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:23:40.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:40.683+0000 7f05cca10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:23:40.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:40.683+0000 7f05cca10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:23:40.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:40.683+0000 7f05cca10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:23:40.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:40.683+0000 7f05cca10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:23:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:40.687+0000 7f05cca10640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:23:43.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:43.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:43.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:43.660+0000 7fed75d16640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:23:43.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:43.660+0000 7fed75d16640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:23:43.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:43.660+0000 7fed75d16640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:23:43.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:43.660+0000 7fed75d16640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:23:43.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:43.660+0000 7fed75d16640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:23:43.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:43.661+0000 7fed75d16640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:23:46.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:23:46.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:46.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:46.627+0000 7f271c0ce640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:23:46.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:46.627+0000 7f271c0ce640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:23:46.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:46.627+0000 7f271c0ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:23:46.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:46.627+0000 7f271c0ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:23:46.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:46.627+0000 7f271c0ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:23:46.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:46.628+0000 7f271c0ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:23:49.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:23:49.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:49.596+0000 7ff2d38f5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:23:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:49.596+0000 7ff2d38f5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:23:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:49.596+0000 7ff2d38f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:23:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:49.596+0000 7ff2d38f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:23:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:49.596+0000 7ff2d38f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:23:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:23:49.597+0000 7ff2d38f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:23:52.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:23:52.463 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:23:52.463 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:52.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:23:52.496 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:52.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:23:52.568 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:52.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:23:52.637 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:52.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:23:52.706 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:52.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:23:52.774 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:52.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:23:52.806 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:52.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:23:52.877 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:52.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:23:52.946 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:52.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:23:53.014 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:53.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:23:53.082 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:53.082 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:23:53.098 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:23:53.098 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:23:53.137 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:23:53.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:23:53.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCqpAZm6XF0DxAAvWxUfj/pfudnoCqr9/hJ6A== 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCtpAZmXvCxJhAAsOOtfNC2uGWg1mPh5fsyqA== 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCwpAZmoFgBJRAAeMPh2BcmUuZTV+I7O7jK2g== 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCzpAZmROVqIxAAVfuk4n/KvqSI0ELWbTn84Q== 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-03-29T11:23:53.181 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC2pAZmk5XqIRAAQCBDOkgJjnOLX6/sE7Mttw== 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCqpAZm7wdeExAAT1gUYiYDCbnJc5cdeZIbpg== 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC8pAZmHs/KKBAAplKq3wAmbJw804F5qEafUg== 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC/pAZmq5NsJxAAkSjEwrC3D0CzHt8qw2bcyQ== 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDCpAZmy7pzJRAApmHjtT41gEdoHn2/brrBow== 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDFpAZmLwabIxAAAmZwNaRj5TOxp0Yqxe58wQ== 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-03-29T11:23:53.182 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCqpAZml4UQFhAA6XN65Km/nzMYvnb0SOh2YA== 2024-03-29T11:23:53.183 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-03-29T11:23:53.183 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCqpAZmOCF3GRAAwDZF6rVlzfUHiDWTnttcjQ== 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCqpAZm6XF0DxAAvWxUfj/pfudnoCqr9/hJ6A== 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCtpAZmXvCxJhAAsOOtfNC2uGWg1mPh5fsyqA== 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCwpAZmoFgBJRAAeMPh2BcmUuZTV+I7O7jK2g== 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCzpAZmROVqIxAAVfuk4n/KvqSI0ELWbTn84Q== 2024-03-29T11:23:53.204 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQC2pAZmk5XqIRAAQCBDOkgJjnOLX6/sE7Mttw== 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCqpAZm7wdeExAAT1gUYiYDCbnJc5cdeZIbpg== 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQC8pAZmHs/KKBAAplKq3wAmbJw804F5qEafUg== 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQC/pAZmq5NsJxAAkSjEwrC3D0CzHt8qw2bcyQ== 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDCpAZmy7pzJRAApmHjtT41gEdoHn2/brrBow== 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDFpAZmLwabIxAAAmZwNaRj5TOxp0Yqxe58wQ== 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-29T11:23:53.205 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCqpAZml4UQFhAA6XN65Km/nzMYvnb0SOh2YA== 2024-03-29T11:23:53.206 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-03-29T11:23:53.206 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCqpAZmOCF3GRAAwDZF6rVlzfUHiDWTnttcjQ== 2024-03-29T11:23:53.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:23:53.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:23:53.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:23:53.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:23:53.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:23:53.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:23:53.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:23:53.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:23:53.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:23:53.900 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:23:53.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:23:53.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:23:54.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:23:54.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:23:54.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:23:54.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:23:54.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:23:54.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:23:54.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:23:54.391 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:54.393 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:54.446 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:23:54.446 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:23:54.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:23:54.449 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:23:54.449 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:23:54.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:23:54.451 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:23:54.451 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:23:54.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:23:54.488 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:23:54.488 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:23:54.488 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:23:54.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:23:54.491 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:23:54.491 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:23:54.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:23:54.493 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:23:54.493 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:23:54.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:23:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.614+0000 7f2eafe3e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.614+0000 7f2eafe3e640 10 monclient: build_initial_monmap 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.615+0000 7f2eafe3e640 10 monclient: monmap: 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:54.615937+0000 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:54.615937+0000 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.615+0000 7f2eafe3e640 1 Processor -- start 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.615+0000 7f2eafe3e640 1 -- start start 2024-03-29T11:23:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.615+0000 7f2eafe3e640 10 monclient: init 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient(hunting): picked mon.noname-c con 0x7f2ea809c870 addr v1:172.21.15.136:6789/0 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient(hunting): picked mon.noname-a con 0x7f2ea80b4360 addr v1:172.21.15.64:6789/0 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eadbda640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2ea809ac40 0x7f2ea80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38840/0 (socket says 172.21.15.64:38840) 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eae3db640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2ea80b4360 0x7f2ea814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40742/0 (socket says 172.21.15.64:40742) 2024-03-29T11:23:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eadbda640 1 -- 172.21.15.64:0/907766625 learned_addr learned my addr 172.21.15.64:0/907766625 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:23:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea809dec0 con 0x7f2ea80b4360 2024-03-29T11:23:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea809da30 con 0x7f2ea809c870 2024-03-29T11:23:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80b39c0 con 0x7f2ea809ac40 2024-03-29T11:23:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:54.616+0000 7f2eafe3e640 10 monclient(hunting): authenticate will time out at 1170.886495s 2024-03-29T11:23:57.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): tick 2024-03-29T11:23:57.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 monclient(hunting): continuing hunt 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.64:6790/0 conn(0x7f2ea809ac40 legacy=0x7f2ea80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2ea809c870 legacy=0x7f2ea813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.64:6789/0 conn(0x7f2ea80b4360 legacy=0x7f2ea814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f2e8c0036e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f2e8c0072d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c00afb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea80b5160 con 0x7f2e8c00afb0 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c00fc80 con 0x7f2e8c0072d0 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c010ee0 con 0x7f2e8c0036e0 2024-03-29T11:23:57.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:57.616+0000 7f2e9f7fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559255179 0 0) 0x7f2ea80b5160 con 0x7f2e8c00afb0 2024-03-29T11:23:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:23:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e94003520 con 0x7f2e8c00afb0 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773843643 0 0) 0x7f2e8c00fc80 con 0x7f2e8c0072d0 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.790+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea80b5160 con 0x7f2e8c0072d0 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3924604989 0 0) 0x7f2ea80b5160 con 0x7f2e8c0072d0 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.64:6790/0 conn(0x7f2e8c0036e0 legacy=0x7f2e8c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.64:6789/0 conn(0x7f2e8c00afb0 legacy=0x7f2e8c00d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 monclient: found mon.noname-c 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:23:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c012140 con 0x7f2e8c0072d0 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ea4001690 con 0x7f2e8c0072d0 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: dump: 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eafe3e640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eafe3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2ead3d9640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2e8c0072d0 legacy=0x7f2e8c009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2ead3d9640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2e8c0072d0 legacy=0x7f2e8c009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2ead3d9640 1 --1- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2e8c0072d0 0x7f2e8c009740 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2ead3d9640 1 --1- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2e8c0072d0 0x7f2e8c009740 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: ms_handle_reset current mon v1:172.21.15.136:6789/0 2024-03-29T11:23:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2e8c0072d0 legacy=0x7f2e8c009740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient(hunting): picked mon.b con 0x7f2e94003cf0 addr v1:172.21.15.136:6789/0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient(hunting): picked mon.c con 0x7f2e8c0036e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient(hunting): picked mon.a con 0x7f2e8c00afb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ea80b5160 con 0x7f2e8c00afb0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2e8c00fc80 con 0x7f2e94003cf0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2e940045b0 con 0x7f2e8c0036e0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.791+0000 7f2eacbd8640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.136:6789/0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074278010 0 0) 0x7f2ea80b5160 con 0x7f2e8c00afb0 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:23:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f2e9400f0c0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1097094905 0 0) 0x7f2e9400f0c0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.64:6790/0 conn(0x7f2e8c0036e0 legacy=0x7f2e94004ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.136:6789/0 conn(0x7f2e94003cf0 legacy=0x7f2e940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 monclient: found mon.a 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f2e94012250 con 0x7f2e8c00afb0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ea0001980 con 0x7f2e8c00afb0 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: dump: 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eafe3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 1 -- 172.21.15.64:0/907766625 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ea0001ae0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eacbd8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eafe3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.792+0000 7f2eafe3e640 4 set_mon_vals no callback set 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 10 monclient: shutdown 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 >> v1:172.21.15.64:6789/0 conn(0x7f2e8c00afb0 legacy=0x7f2e9400d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 shutdown_connections 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 >> 172.21.15.64:0/907766625 conn(0x7f2ea8066b00 msgr2=0x7f2ea8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 shutdown_connections 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.793+0000 7f2eafe3e640 1 -- 172.21.15.64:0/907766625 wait complete. 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.794+0000 7f2eafe3e640 10 monclient: build_initial_monmap 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.794+0000 7f2eafe3e640 10 monclient: monmap: 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T11:23:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 1 librados: starting msgr at 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 1 librados: starting objecter 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 1 Processor -- start 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 1 -- start start 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 1 librados: setting wanted keys 2024-03-29T11:23:59.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 1 librados: calling monclient init 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.795+0000 7f2eafe3e640 10 monclient: init 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c0036e0 addr v1:172.21.15.64:6789/0 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 10 monclient(hunting): picked mon.noname-c con 0x7f2e8c00afb0 addr v1:172.21.15.64:6790/0 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eadbda640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2e8c0036e0 0x7f2ea8138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55754/0 (socket says 172.21.15.64:55754) 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eadbda640 1 -- 172.21.15.64:0/3494795513 learned_addr learned my addr 172.21.15.64:0/3494795513 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f2e8c0036e0 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2e9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f2e8c00afb0 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2e9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 10 monclient(hunting): picked mon.noname-b con 0x7f2e94003cf0 addr v1:172.21.15.136:6789/0 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea81396a0 con 0x7f2e8c0036e0 2024-03-29T11:23:59.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea8139e50 con 0x7f2e94003cf0 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea8139930 con 0x7f2e8c00afb0 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2eafe3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f2e94003cf0 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.796+0000 7f2e9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2eae3db640 1 -- 172.21.15.64:0/3494795513 >> v1:172.21.15.136:6789/0 conn(0x7f2e94003cf0 legacy=0x7f2ea81da610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2eae3db640 1 -- 172.21.15.64:0/3494795513 >> v1:172.21.15.136:6789/0 conn(0x7f2e94003cf0 legacy=0x7f2ea81da610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2eae3db640 1 --1- 172.21.15.64:0/3494795513 >> v1:172.21.15.136:6789/0 conn(0x7f2e94003cf0 0x7f2ea81da610 :-1 s=OPENED pgs=15 cs=1 l=1).handle_message read tag failed 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2eae3db640 1 --1- 172.21.15.64:0/3494795513 >> v1:172.21.15.136:6789/0 conn(0x7f2e94003cf0 0x7f2ea81da610 :-1 s=OPENED pgs=15 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2eafe3e640 10 monclient(hunting): authenticate will time out at 1176.066940s 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.136:6789/0 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518410040 0 0) 0x7f2ea8139930 con 0x7f2e8c00afb0 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c003540 con 0x7f2e8c00afb0 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1957328946 0 0) 0x7f2e940045b0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ea8139930 con 0x7f2e8c00afb0 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ea4001870 con 0x7f2e8c00afb0 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:23:59.789 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.797+0000 7f2e9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166005984 0 0) 0x7f2ea81396a0 con 0x7f2e8c0036e0 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient(hunting): my global_id is 4107 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e940045b0 con 0x7f2e8c0036e0 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 315566579 0 0) 0x7f2ea8139930 con 0x7f2e8c00afb0 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 >> v1:172.21.15.136:6789/0 conn(0x7f2e94003cf0 legacy=0x7f2ea81da610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 >> v1:172.21.15.64:6789/0 conn(0x7f2e8c0036e0 legacy=0x7f2ea8138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 monclient: found mon.c 2024-03-29T11:23:59.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8138ce0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 10 monclient: _renew_subs 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea8136690 con 0x7f2e8c00afb0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 10 client.4106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 10 monclient: _renew_subs 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea80b84a0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2eafe3e640 1 librados: init done 2024-03-29T11:23:59.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ea4001cc0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ea4003790 con 0x7f2e8c00afb0 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 monclient: dump: 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:23:59.792 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2ea4003d00 con 0x7f2e8c00afb0 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 117268284 0 0) 0x7f2ea4006350 con 0x7f2e8c00afb0 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 10 client.4106.objecter ms_dispatch 0x7f2ea8000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 3 client.4106.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 3 client.4106.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.798+0000 7f2e9e7fc640 20 client.4106.objecter dump_active .. 0 homeless 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.799+0000 7f2e9ffff640 4 set_mon_vals no callback set 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.799+0000 7f2eafe3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.800+0000 7f2eafe3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.800+0000 7f2eafe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.800+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90005080 con 0x7f2e8c00afb0 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.802+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2ea40057a0 con 0x7f2e8c00afb0 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.802+0000 7f2e9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.802+0000 7f2e9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.873+0000 7f2eafe3e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:23:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.873+0000 7f2eafe3e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:23:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.873+0000 7f2eafe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:23:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.873+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2e90005330 con 0x7f2e8c00afb0 2024-03-29T11:23:59.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.878+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/3494795513 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f2ea4005a20 con 0x7f2e8c00afb0 2024-03-29T11:23:59.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.878+0000 7f2e9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:23:59.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.878+0000 7f2e9e7fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:23:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 10 librados: watch_flush enter 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 10 librados: watch_flush exit 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 20 client.4106.objecter shutdown clearing up homeless session... 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 10 client.4106.objecter successfully canceled tick 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 10 monclient: shutdown 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 >> v1:172.21.15.64:6790/0 conn(0x7f2e8c00afb0 legacy=0x7f2ea8135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 shutdown_connections 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 >> 172.21.15.64:0/3494795513 conn(0x7f2ea8066b00 msgr2=0x7f2ea8069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 shutdown_connections 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 1 -- 172.21.15.64:0/3494795513 wait complete. 2024-03-29T11:23:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:23:59.879+0000 7f2eafe3e640 1 librados: shutdown 2024-03-29T11:23:59.870 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:23:59.871 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:23:59.871 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:59.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:23:59.898 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:59.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:23:59.965 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:23:59.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:24:00.032 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T11:24:00.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:24:00.099 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:24:00.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:24:00.126 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:24:00.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:24:00.194 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:24:00.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:24:00.254 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:00.274+0000 7f4941e2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:24:00.261 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T11:24:00.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:24:00.268 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:00.288+0000 7fdf4d3f1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:24:00.310 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:00.330+0000 7f4941e2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:24:00.324 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:00.344+0000 7fdf4d3f1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:24:00.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7 0 2024-03-29T11:24:00.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.487+0000 7f160eb8f640 10 monclient: get_monmap_and_config 2024-03-29T11:24:00.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.487+0000 7f160eb8f640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.487+0000 7f160eb8f640 10 monclient: monmap: 2024-03-29T11:24:00.468 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:00.468 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:00.488930+0000 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:00.488930+0000 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:00.469 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.488+0000 7f160eb8f640 1 Processor -- start 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.488+0000 7f160eb8f640 1 -- start start 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.488+0000 7f160eb8f640 10 monclient: init 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient(hunting): picked mon.noname-b con 0x7f160809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient(hunting): picked mon.noname-a con 0x7f160809c870 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient(hunting): picked mon.noname-c con 0x7f16080b4360 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:00.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160809dec0 con 0x7f160809c870 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160809da30 con 0x7f16080b4360 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16080b39c0 con 0x7f160809ac40 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160eb8f640 10 monclient(hunting): authenticate will time out at 774.661143s 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160d12c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f16080b4360 0x7f160814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39760/0 (socket says 172.21.15.136:39760) 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f160d12c640 1 -- 172.21.15.136:0/1241916949 learned_addr learned my addr 172.21.15.136:0/1241916949 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522784029 0 0) 0x7f160809da30 con 0x7f16080b4360 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f8003520 con 0x7f16080b4360 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947791914 0 0) 0x7f15f4001110 con 0x7f160809c870 2024-03-29T11:24:00.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160809da30 con 0x7f160809c870 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281739593 0 0) 0x7f16080b39c0 con 0x7f160809ac40 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4001110 con 0x7f160809ac40 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242395176 0 0) 0x7f15f8003520 con 0x7f16080b4360 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 >> v1:172.21.15.64:6790/0 conn(0x7f160809ac40 legacy=0x7f16080b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 >> v1:172.21.15.64:6789/0 conn(0x7f160809c870 legacy=0x7f160813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16080b5160 con 0x7f16080b4360 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.489+0000 7f16077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1600001a10 con 0x7f16080b4360 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: dump: 2024-03-29T11:24:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1600001bf0 con 0x7f16080b4360 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 1 -- 172.21.15.136:0/1241916949 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1600002000 con 0x7f16080b4360 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 4 set_mon_vals no callback set 2024-03-29T11:24:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f16077fe640 10 monclient: dump: 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 10 monclient: shutdown 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 1 -- 172.21.15.136:0/1241916949 >> v1:172.21.15.136:6789/0 conn(0x7f16080b4360 legacy=0x7f160814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 1 -- 172.21.15.136:0/1241916949 shutdown_connections 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 1 -- 172.21.15.136:0/1241916949 >> 172.21.15.136:0/1241916949 conn(0x7f1608066b00 msgr2=0x7f1608066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 1 -- 172.21.15.136:0/1241916949 shutdown_connections 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.490+0000 7f160eb8f640 1 -- 172.21.15.136:0/1241916949 wait complete. 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 10 monclient: monmap: 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:00.485 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 1 librados: starting msgr at 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 1 librados: starting objecter 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 1 Processor -- start 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 1 -- start start 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 1 librados: setting wanted keys 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 1 librados: calling monclient init 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.491+0000 7f160eb8f640 10 monclient: init 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient(hunting): picked mon.noname-c con 0x7f160809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient(hunting): picked mon.noname-a con 0x7f16081406b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient(hunting): picked mon.noname-b con 0x7f16080b07d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:00.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1608142220 con 0x7f16081406b0 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16081399e0 con 0x7f16080b07d0 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1608143990 con 0x7f160809ac40 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160eb8f640 10 monclient(hunting): authenticate will time out at 774.664397s 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160d12c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f16080b07d0 0x7f16081392d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39774/0 (socket says 172.21.15.136:39774) 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f160d12c640 1 -- 172.21.15.136:0/1755517270 learned_addr learned my addr 172.21.15.136:0/1755517270 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 client.?.objecter ms_handle_connect 0x7f16080b07d0 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 client.?.objecter ms_handle_connect 0x7f16081406b0 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 client.?.objecter ms_handle_connect 0x7f160809ac40 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744500923 0 0) 0x7f16081399e0 con 0x7f16080b07d0 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec004eb0 con 0x7f16080b07d0 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286509953 0 0) 0x7f1608142220 con 0x7f16081406b0 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16081399e0 con 0x7f16081406b0 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854601468 0 0) 0x7f1608143990 con 0x7f160809ac40 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:24:00.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.492+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1608142220 con 0x7f160809ac40 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3220299554 0 0) 0x7f15ec004eb0 con 0x7f16080b07d0 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1608143990 con 0x7f16080b07d0 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1600001f40 con 0x7f16080b07d0 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.489 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 442065117 0 0) 0x7f16081399e0 con 0x7f16081406b0 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15ec004eb0 con 0x7f16081406b0 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f15f40018e0 con 0x7f16081406b0 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.490 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461377463 0 0) 0x7f1608142220 con 0x7f160809ac40 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16081399e0 con 0x7f160809ac40 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f15fc0017f0 con 0x7f160809ac40 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.491 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 464729776 0 0) 0x7f1608143990 con 0x7f16080b07d0 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 >> v1:172.21.15.64:6790/0 conn(0x7f160809ac40 legacy=0x7f160813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 >> v1:172.21.15.64:6789/0 conn(0x7f16081406b0 legacy=0x7f1608138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 monclient: found mon.b 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160813a1b0 con 0x7f16080b07d0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 10 monclient: _renew_subs 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1608141330 con 0x7f16080b07d0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 10 monclient: _renew_subs 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16081426b0 con 0x7f16080b07d0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f160eb8f640 1 librados: init done 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1600004e10 con 0x7f16080b07d0 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1600005120 con 0x7f16080b07d0 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f1606ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 monclient: dump: 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1600005610 con 0x7f16080b07d0 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1733892266 0 0) 0x7f1600006ae0 con 0x7f16080b07d0 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 10 client.4117.objecter ms_dispatch 0x7f1608000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.493+0000 7f16057fa640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.494+0000 7f160eb8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.494+0000 7f160eb8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.494+0000 7f160eb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.494+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e8005080 con 0x7f16080b07d0 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.496+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1600005b00 con 0x7f16080b07d0 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.496+0000 7f16057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.496+0000 7f16057fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:00.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.567+0000 7f160eb8f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7", "id": 0}] 2024-03-29T11:24:00.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.567+0000 7f160eb8f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7", "id": 0}] 2024-03-29T11:24:00.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.567+0000 7f160eb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.567+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7", "id": 0} v 0) v1 -- 0x7f15e80058b0 con 0x7f16080b07d0 2024-03-29T11:24:00.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.570+0000 7f16057fa640 1 -- 172.21.15.136:0/1755517270 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2778752290 0 732707515) 0x7f1600005d70 con 0x7f16080b07d0 2024-03-29T11:24:00.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.570+0000 7f16057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7", "id": 0}] 2024-03-29T11:24:00.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.570+0000 7f16057fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:00.551 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 10 librados: watch_flush enter 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 10 librados: watch_flush exit 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 10 monclient: shutdown 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 >> v1:172.21.15.136:6789/0 conn(0x7f16080b07d0 legacy=0x7f16081392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 shutdown_connections 2024-03-29T11:24:00.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 >> 172.21.15.136:0/1755517270 conn(0x7f1608066b00 msgr2=0x7f160813b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 shutdown_connections 2024-03-29T11:24:00.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 1 -- 172.21.15.136:0/1755517270 wait complete. 2024-03-29T11:24:00.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.571+0000 7f160eb8f640 1 librados: shutdown 2024-03-29T11:24:00.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 41cd56e7-b77b-465b-be8e-86e47bd7e82e 1 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.681+0000 7fedb6979640 10 monclient: get_monmap_and_config 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.681+0000 7fedb6979640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.681+0000 7fedb6979640 10 monclient: monmap: 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:00.682556+0000 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:00.682556+0000 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:00.662 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:00.663 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:00.663 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 1 Processor -- start 2024-03-29T11:24:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 1 -- start start 2024-03-29T11:24:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient: init 2024-03-29T11:24:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient(hunting): picked mon.noname-b con 0x7fedb009ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient(hunting): picked mon.noname-a con 0x7fedb009c870 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient(hunting): picked mon.noname-c con 0x7fedb00b4360 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb009dec0 con 0x7fedb009c870 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb009da30 con 0x7fedb00b4360 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb00b39c0 con 0x7fedb009ac40 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.682+0000 7fedb6979640 10 monclient(hunting): authenticate will time out at 774.854723s 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb4f16640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fedb00b4360 0x7fedb014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39780/0 (socket says 172.21.15.136:39780) 2024-03-29T11:24:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb4f16640 1 -- 172.21.15.136:0/1103802271 learned_addr learned my addr 172.21.15.136:0/1103802271 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805669933 0 0) 0x7fedb009da30 con 0x7fedb00b4360 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed98003520 con 0x7fedb00b4360 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388810091 0 0) 0x7feda4001040 con 0x7fedb009c870 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb009da30 con 0x7fedb009c870 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421483831 0 0) 0x7fedb00b39c0 con 0x7fedb009ac40 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4001040 con 0x7fedb009ac40 2024-03-29T11:24:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 273992895 0 0) 0x7fed98003520 con 0x7fedb00b4360 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 >> v1:172.21.15.64:6790/0 conn(0x7fedb009ac40 legacy=0x7fedb00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 >> v1:172.21.15.64:6789/0 conn(0x7fedb009c870 legacy=0x7fedb013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 monclient: found mon.noname-c 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00b5160 con 0x7fedb00b4360 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb6979640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7feda8001a10 con 0x7fedb00b4360 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb6979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: dump: 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb6979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda8001bf0 con 0x7fedb00b4360 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb6979640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 1 -- 172.21.15.136:0/1103802271 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7feda8002000 con 0x7fedb00b4360 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedb6979640 4 set_mon_vals no callback set 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.683+0000 7fedaeffd640 10 monclient: dump: 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 10 monclient: shutdown 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 1 -- 172.21.15.136:0/1103802271 >> v1:172.21.15.136:6789/0 conn(0x7fedb00b4360 legacy=0x7fedb014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 1 -- 172.21.15.136:0/1103802271 shutdown_connections 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 1 -- 172.21.15.136:0/1103802271 >> 172.21.15.136:0/1103802271 conn(0x7fedb0066b00 msgr2=0x7fedb0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 1 -- 172.21.15.136:0/1103802271 shutdown_connections 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 1 -- 172.21.15.136:0/1103802271 wait complete. 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.684+0000 7fedb6979640 10 monclient: monmap: 2024-03-29T11:24:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:00.683 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:00.683 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:00.683 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 librados: starting msgr at 2024-03-29T11:24:00.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 librados: starting objecter 2024-03-29T11:24:00.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 Processor -- start 2024-03-29T11:24:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 -- start start 2024-03-29T11:24:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 librados: setting wanted keys 2024-03-29T11:24:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 librados: calling monclient init 2024-03-29T11:24:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient: init 2024-03-29T11:24:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient(hunting): picked mon.noname-c con 0x7fedb009ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient(hunting): picked mon.noname-b con 0x7fedb01d7cd0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient(hunting): picked mon.noname-a con 0x7fedb00ad1e0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb01d7130 con 0x7fedb00ad1e0 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb0139b10 con 0x7fedb01d7cd0 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb013ffa0 con 0x7fedb009ac40 2024-03-29T11:24:00.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedb6979640 10 monclient(hunting): authenticate will time out at 774.857744s 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.685+0000 7fedaf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fedb01d7cd0 0x7fedb0138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39792/0 (socket says 172.21.15.136:39792) 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedaf7fe640 1 -- 172.21.15.136:0/182391057 learned_addr learned my addr 172.21.15.136:0/182391057 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 client.?.objecter ms_handle_connect 0x7fedb01d7cd0 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510802247 0 0) 0x7fedb0139b10 con 0x7fedb01d7cd0 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9c004eb0 con 0x7fedb01d7cd0 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 client.?.objecter ms_handle_connect 0x7fedb009ac40 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 client.?.objecter ms_handle_connect 0x7fedb00ad1e0 2024-03-29T11:24:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268584361 0 0) 0x7fedb013ffa0 con 0x7fedb009ac40 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0139b10 con 0x7fedb009ac40 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446757304 0 0) 0x7fedb01d7130 con 0x7fedb00ad1e0 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:24:00.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb013ffa0 con 0x7fedb00ad1e0 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964398992 0 0) 0x7fed9c004eb0 con 0x7fedb01d7cd0 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedb01d7130 con 0x7fedb01d7cd0 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7feda4001950 con 0x7fedb01d7cd0 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): dump: 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.689 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 989347512 0 0) 0x7fedb0139b10 con 0x7fedb009ac40 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed9c004eb0 con 0x7fedb009ac40 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7feda00018b0 con 0x7fedb009ac40 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): dump: 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.690 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1933529653 0 0) 0x7fedb013ffa0 con 0x7fedb00ad1e0 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedb0139b10 con 0x7fedb00ad1e0 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7feda8001d60 con 0x7fedb00ad1e0 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient(hunting): dump: 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3729418754 0 0) 0x7fedb01d7130 con 0x7fedb01d7cd0 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 >> v1:172.21.15.64:6790/0 conn(0x7fedb009ac40 legacy=0x7fedb01d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 >> v1:172.21.15.64:6789/0 conn(0x7fedb00ad1e0 legacy=0x7fedb0139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 monclient: found mon.b 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00ae520 con 0x7fedb01d7cd0 2024-03-29T11:24:00.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 10 monclient: _renew_subs 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb00ae800 con 0x7fedb01d7cd0 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 10 monclient: _renew_subs 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb0142b70 con 0x7fedb01d7cd0 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedb6979640 1 librados: init done 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda4003610 con 0x7fedb01d7cd0 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7feda4003920 con 0x7fedb01d7cd0 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedae7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 10 monclient: dump: 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.694 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.697 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.697 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.697 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.686+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feda4003e90 con 0x7fedb01d7cd0 2024-03-29T11:24:00.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 741510870 0 0) 0x7feda4006560 con 0x7fedb01d7cd0 2024-03-29T11:24:00.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedacff9640 10 client.4123.objecter ms_dispatch 0x7fedb0000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedacff9640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedacff9640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedacff9640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedb6979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedb6979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedb6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.687+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed90005080 con 0x7fedb01d7cd0 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.689+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feda4004320 con 0x7fedb01d7cd0 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.689+0000 7fedacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.689+0000 7fedacff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:00.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.762+0000 7fedb6979640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "41cd56e7-b77b-465b-be8e-86e47bd7e82e", "id": 1}] 2024-03-29T11:24:00.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.762+0000 7fedb6979640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "41cd56e7-b77b-465b-be8e-86e47bd7e82e", "id": 1}] 2024-03-29T11:24:00.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.762+0000 7fedb6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.762+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "41cd56e7-b77b-465b-be8e-86e47bd7e82e", "id": 1} v 0) v1 -- 0x7fed900058b0 con 0x7fedb01d7cd0 2024-03-29T11:24:00.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.769+0000 7fedacff9640 1 -- 172.21.15.136:0/182391057 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41cd56e7-b77b-465b-be8e-86e47bd7e82e", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 232187410 0 940483276) 0x7feda4005890 con 0x7fedb01d7cd0 2024-03-29T11:24:00.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "41cd56e7-b77b-465b-be8e-86e47bd7e82e", "id": 1}] 2024-03-29T11:24:00.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedacff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:00.750 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-29T11:24:00.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 10 librados: watch_flush enter 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 10 librados: watch_flush exit 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 10 monclient: shutdown 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 >> v1:172.21.15.136:6789/0 conn(0x7fedb01d7cd0 legacy=0x7fedb0138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 shutdown_connections 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.770+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 >> 172.21.15.136:0/182391057 conn(0x7fedb0066b00 msgr2=0x7fedb00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.771+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 shutdown_connections 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.771+0000 7fedb6979640 1 -- 172.21.15.136:0/182391057 wait complete. 2024-03-29T11:24:00.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.771+0000 7fedb6979640 1 librados: shutdown 2024-03-29T11:24:00.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 5e2d90e3-43b4-415d-ab63-6ca41b99d92c 2 2024-03-29T11:24:00.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.883+0000 7f8d1ab7b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:00.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.883+0000 7f8d1ab7b640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.883+0000 7f8d1ab7b640 10 monclient: monmap: 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:00.884085+0000 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:00.884085+0000 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:00.864 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.883+0000 7f8d1ab7b640 1 Processor -- start 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.883+0000 7f8d1ab7b640 1 -- start start 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.883+0000 7f8d1ab7b640 10 monclient: init 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient(hunting): picked mon.noname-a con 0x7f8d1409ac40 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient(hunting): picked mon.noname-c con 0x7f8d1409c870 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient(hunting): picked mon.noname-b con 0x7f8d140b4360 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:00.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1409dec0 con 0x7f8d1409ac40 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1409da30 con 0x7f8d1409c870 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d140b39c0 con 0x7f8d140b4360 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d1ab7b640 10 monclient(hunting): authenticate will time out at 775.056313s 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d13fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8d1409c870 0x7f8d1413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39808/0 (socket says 172.21.15.136:39808) 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d13fff640 1 -- 172.21.15.136:0/2750704226 learned_addr learned my addr 172.21.15.136:0/2750704226 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21732215 0 0) 0x7f8d1409da30 con 0x7f8d1409c870 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d137fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.884+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d04003520 con 0x7f8d1409c870 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683566824 0 0) 0x7f8d140b39c0 con 0x7f8d140b4360 2024-03-29T11:24:00.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1409da30 con 0x7f8d140b4360 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656886227 0 0) 0x7f8d08001040 con 0x7f8d1409ac40 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d140b39c0 con 0x7f8d1409ac40 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2599653884 0 0) 0x7f8d04003520 con 0x7f8d1409c870 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 >> v1:172.21.15.64:6790/0 conn(0x7f8d140b4360 legacy=0x7f8d1414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 >> v1:172.21.15.64:6789/0 conn(0x7f8d1409ac40 legacy=0x7f8d140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140b5160 con 0x7f8d1409c870 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8d00001610 con 0x7f8d1409c870 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: dump: 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.871 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d00001770 con 0x7f8d1409c870 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 1 -- 172.21.15.136:0/2750704226 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8d00001bc0 con 0x7f8d1409c870 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 4 set_mon_vals no callback set 2024-03-29T11:24:00.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d137fe640 10 monclient: dump: 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 10 monclient: shutdown 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2750704226 >> v1:172.21.15.136:6789/0 conn(0x7f8d1409c870 legacy=0x7f8d1413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2750704226 shutdown_connections 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2750704226 >> 172.21.15.136:0/2750704226 conn(0x7f8d14066b00 msgr2=0x7f8d14066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2750704226 shutdown_connections 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.885+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2750704226 wait complete. 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.886+0000 7f8d1ab7b640 10 monclient: build_initial_monmap 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.886+0000 7f8d1ab7b640 10 monclient: monmap: 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:00.874 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 1 librados: starting msgr at 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 1 librados: starting objecter 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 1 Processor -- start 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 1 -- start start 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 1 librados: setting wanted keys 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 1 librados: calling monclient init 2024-03-29T11:24:00.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.887+0000 7f8d1ab7b640 10 monclient: init 2024-03-29T11:24:00.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:00.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:00.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient(hunting): picked mon.noname-b con 0x7f8d1409ac40 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:00.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient(hunting): picked mon.noname-a con 0x7f8d140add50 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:00.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient(hunting): picked mon.noname-c con 0x7f8d141dac90 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:00.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d14120390 con 0x7f8d140add50 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d141da770 con 0x7f8d1409ac40 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1413a630 con 0x7f8d141dac90 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d1ab7b640 10 monclient(hunting): authenticate will time out at 775.060399s 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d18917640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8d1409ac40 0x7f8d140b1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39812/0 (socket says 172.21.15.136:39812) 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d18917640 1 -- 172.21.15.136:0/2072875465 learned_addr learned my addr 172.21.15.136:0/2072875465 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 client.?.objecter ms_handle_connect 0x7f8d1409ac40 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 client.?.objecter ms_handle_connect 0x7f8d140add50 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 client.?.objecter ms_handle_connect 0x7f8d141dac90 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597472936 0 0) 0x7f8d141da770 con 0x7f8d1409ac40 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf8004eb0 con 0x7f8d1409ac40 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941256929 0 0) 0x7f8d1413a630 con 0x7f8d141dac90 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d141da770 con 0x7f8d141dac90 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349063940 0 0) 0x7f8d14120390 con 0x7f8d140add50 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:24:00.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.888+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1413a630 con 0x7f8d140add50 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821360981 0 0) 0x7f8cf8004eb0 con 0x7f8d1409ac40 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d14120390 con 0x7f8d1409ac40 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8d08001790 con 0x7f8d1409ac40 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 208505929 0 0) 0x7f8d141da770 con 0x7f8d141dac90 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cf8004eb0 con 0x7f8d141dac90 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8d0c001b90 con 0x7f8d141dac90 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 531842959 0 0) 0x7f8d1413a630 con 0x7f8d140add50 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d141da770 con 0x7f8d140add50 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8d000018e0 con 0x7f8d140add50 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient(hunting): dump: 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.885 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46360708 0 0) 0x7f8d14120390 con 0x7f8d1409ac40 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 >> v1:172.21.15.64:6790/0 conn(0x7f8d141dac90 legacy=0x7f8d14139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 >> v1:172.21.15.64:6789/0 conn(0x7f8d140add50 legacy=0x7f8d140b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 monclient: found mon.b 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1413a7c0 con 0x7f8d1409ac40 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 10 monclient: _renew_subs 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d141db140 con 0x7f8d1409ac40 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 10 monclient: _renew_subs 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d141e3980 con 0x7f8d1409ac40 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d1ab7b640 1 librados: init done 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d08001d00 con 0x7f8d1409ac40 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8d080037b0 con 0x7f8d1409ac40 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d12ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 monclient: dump: 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8d08003d20 con 0x7f8d1409ac40 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3339086526 0 0) 0x7f8d08006580 con 0x7f8d1409ac40 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 10 client.4129.objecter ms_dispatch 0x7f8d14000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.889+0000 7f8d117fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.890+0000 7f8d1ab7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.890+0000 7f8d1ab7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.890+0000 7f8d1ab7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.890+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf4005080 con 0x7f8d1409ac40 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.894+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8d080042c0 con 0x7f8d1409ac40 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.894+0000 7f8d117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.894+0000 7f8d117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:00.889 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:00.896+0000 7f4941e2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:24:00.890 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:00.896+0000 7fdf4d3f1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:24:00.932 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:00.952+0000 7f4941e2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:24:00.933 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:00.952+0000 7fdf4d3f1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:24:00.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.966+0000 7f8d1ab7b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5e2d90e3-43b4-415d-ab63-6ca41b99d92c", "id": 2}] 2024-03-29T11:24:00.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.966+0000 7f8d1ab7b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5e2d90e3-43b4-415d-ab63-6ca41b99d92c", "id": 2}] 2024-03-29T11:24:00.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.966+0000 7f8d1ab7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:00.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.966+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5e2d90e3-43b4-415d-ab63-6ca41b99d92c", "id": 2} v 0) v1 -- 0x7f8cf40058b0 con 0x7f8d1409ac40 2024-03-29T11:24:00.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d117fa640 1 -- 172.21.15.136:0/2072875465 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5e2d90e3-43b4-415d-ab63-6ca41b99d92c", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 231620423 0 216599125) 0x7f8d08006160 con 0x7f8d1409ac40 2024-03-29T11:24:00.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5e2d90e3-43b4-415d-ab63-6ca41b99d92c", "id": 2}] 2024-03-29T11:24:00.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:00.955 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-03-29T11:24:00.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d1ab7b640 10 librados: watch_flush enter 2024-03-29T11:24:00.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d1ab7b640 10 librados: watch_flush exit 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d1ab7b640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.975+0000 7f8d1ab7b640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 10 monclient: shutdown 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 >> v1:172.21.15.136:6789/0 conn(0x7f8d1409ac40 legacy=0x7f8d140b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 shutdown_connections 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 >> 172.21.15.136:0/2072875465 conn(0x7f8d14066b00 msgr2=0x7f8d1413c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 shutdown_connections 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 1 -- 172.21.15.136:0/2072875465 wait complete. 2024-03-29T11:24:00.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:00.976+0000 7f8d1ab7b640 1 librados: shutdown 2024-03-29T11:24:00.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new aaade9b8-7f3c-404e-98c9-c195e943b41a 3 2024-03-29T11:24:00.979 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:00.999+0000 7fdf4d3f1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:24:00.981 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.000+0000 7f4941e2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:24:01.036 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.055+0000 7fdf4d3f1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:24:01.038 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.058+0000 7f4941e2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:24:01.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.086+0000 7faab5b32640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.086+0000 7faab5b32640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.086+0000 7faab5b32640 10 monclient: monmap: 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:01.087200+0000 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:01.087200+0000 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:01.067 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.086+0000 7faab5b32640 1 Processor -- start 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.086+0000 7faab5b32640 1 -- start start 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.086+0000 7faab5b32640 10 monclient: init 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient(hunting): picked mon.noname-b con 0x7faab009ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient(hunting): picked mon.noname-c con 0x7faab009c870 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient(hunting): picked mon.noname-a con 0x7faab00b4360 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab009dec0 con 0x7faab00b4360 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab009da30 con 0x7faab009c870 2024-03-29T11:24:01.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab00b39c0 con 0x7faab009ac40 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faab5b32640 10 monclient(hunting): authenticate will time out at 775.259395s 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaaeffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7faab009c870 0x7faab013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39826/0 (socket says 172.21.15.136:39826) 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaaeffd640 1 -- 172.21.15.136:0/4291025924 learned_addr learned my addr 172.21.15.136:0/4291025924 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098515661 0 0) 0x7faab009da30 con 0x7faab009c870 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaae7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.087+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa98003520 con 0x7faab009c870 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938991222 0 0) 0x7faab00b39c0 con 0x7faab009ac40 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab009da30 con 0x7faab009ac40 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523936223 0 0) 0x7faaa8001490 con 0x7faab00b4360 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab00b39c0 con 0x7faab00b4360 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143781955 0 0) 0x7faa98003520 con 0x7faab009c870 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 >> v1:172.21.15.64:6790/0 conn(0x7faab009ac40 legacy=0x7faab00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 >> v1:172.21.15.64:6789/0 conn(0x7faab00b4360 legacy=0x7faab014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab00b5160 con 0x7faab009c870 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7faaa4001690 con 0x7faab009c870 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: dump: 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaa4001870 con 0x7faab009c870 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 4 set_mon_vals no callback set 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 1 -- 172.21.15.136:0/4291025924 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7faaa4001c80 con 0x7faab009c870 2024-03-29T11:24:01.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faaae7fc640 10 monclient: dump: 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 10 monclient: shutdown 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 1 -- 172.21.15.136:0/4291025924 >> v1:172.21.15.136:6789/0 conn(0x7faab009c870 legacy=0x7faab013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 1 -- 172.21.15.136:0/4291025924 shutdown_connections 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 1 -- 172.21.15.136:0/4291025924 >> 172.21.15.136:0/4291025924 conn(0x7faab0066b00 msgr2=0x7faab0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 1 -- 172.21.15.136:0/4291025924 shutdown_connections 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.088+0000 7faab5b32640 1 -- 172.21.15.136:0/4291025924 wait complete. 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.089+0000 7faab5b32640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.089+0000 7faab5b32640 10 monclient: monmap: 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:01.082 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.089+0000 7faab5b32640 1 librados: starting msgr at 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.089+0000 7faab5b32640 1 librados: starting objecter 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.089+0000 7faab5b32640 1 Processor -- start 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 1 -- start start 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 1 librados: setting wanted keys 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 1 librados: calling monclient init 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient: init 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient(hunting): picked mon.noname-b con 0x7faab009ac40 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient(hunting): picked mon.noname-a con 0x7faab0141210 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient(hunting): picked mon.noname-c con 0x7faab013ffa0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab0142300 con 0x7faab0141210 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab01425e0 con 0x7faab009ac40 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab01db510 con 0x7faab013ffa0 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faab5b32640 10 monclient(hunting): authenticate will time out at 775.262411s 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faaaf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7faab009ac40 0x7faab01dac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39834/0 (socket says 172.21.15.136:39834) 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faaaf7fe640 1 -- 172.21.15.136:0/1206935015 learned_addr learned my addr 172.21.15.136:0/1206935015 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 client.?.objecter ms_handle_connect 0x7faab009ac40 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 client.?.objecter ms_handle_connect 0x7faab0141210 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447147557 0 0) 0x7faab01425e0 con 0x7faab009ac40 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c004eb0 con 0x7faab009ac40 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 client.?.objecter ms_handle_connect 0x7faab013ffa0 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561405306 0 0) 0x7faab0142300 con 0x7faab0141210 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab01425e0 con 0x7faab0141210 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948779424 0 0) 0x7faab01db510 con 0x7faab013ffa0 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.090+0000 7faa97fff640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab0142300 con 0x7faab013ffa0 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041136063 0 0) 0x7faa9c004eb0 con 0x7faab009ac40 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faab01db510 con 0x7faab009ac40 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7faaa0001a30 con 0x7faab009ac40 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): dump: 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715210504 0 0) 0x7faab01425e0 con 0x7faab0141210 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa9c004eb0 con 0x7faab0141210 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7faaa4001b80 con 0x7faab0141210 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient(hunting): dump: 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 17130357 0 0) 0x7faab01db510 con 0x7faab009ac40 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 >> v1:172.21.15.64:6790/0 conn(0x7faab013ffa0 legacy=0x7faab0140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 >> v1:172.21.15.64:6789/0 conn(0x7faab0141210 legacy=0x7faab00b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 monclient: found mon.b 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab0140b20 con 0x7faab009ac40 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 10 monclient: _renew_subs 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab00b17a0 con 0x7faab009ac40 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 10 monclient: _renew_subs 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faab01e3f60 con 0x7faab009ac40 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaa0001cf0 con 0x7faab009ac40 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faab5b32640 1 librados: init done 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7faaa0003a60 con 0x7faab009ac40 2024-03-29T11:24:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 monclient: dump: 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faaa0004010 con 0x7faab009ac40 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faaadffb640 4 set_mon_vals no callback set 2024-03-29T11:24:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 641585279 0 0) 0x7faaa00042c0 con 0x7faab009ac40 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 10 client.4135.objecter ms_dispatch 0x7faab0000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.091+0000 7faa97fff640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.092+0000 7faab5b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.092+0000 7faab5b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.092+0000 7faab5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.092+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab01db350 con 0x7faab009ac40 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.094+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faaa00057c0 con 0x7faab009ac40 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.094+0000 7faa97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.094+0000 7faa97fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.092 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.104+0000 7f4941e2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:24:01.092 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.099+0000 7fdf4d3f1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:24:01.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.167+0000 7faab5b32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aaade9b8-7f3c-404e-98c9-c195e943b41a", "id": 3}] 2024-03-29T11:24:01.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.167+0000 7faab5b32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aaade9b8-7f3c-404e-98c9-c195e943b41a", "id": 3}] 2024-03-29T11:24:01.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.167+0000 7faab5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.167+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aaade9b8-7f3c-404e-98c9-c195e943b41a", "id": 3} v 0) v1 -- 0x7faab009b020 con 0x7faab009ac40 2024-03-29T11:24:01.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.174+0000 7faa97fff640 1 -- 172.21.15.136:0/1206935015 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aaade9b8-7f3c-404e-98c9-c195e943b41a", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1295010813 0 525046306) 0x7faaa0005a60 con 0x7faab009ac40 2024-03-29T11:24:01.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.174+0000 7faa97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aaade9b8-7f3c-404e-98c9-c195e943b41a", "id": 3}] 2024-03-29T11:24:01.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.174+0000 7faa97fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.155 INFO:teuthology.orchestra.run.smithi136.stdout:3 2024-03-29T11:24:01.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 10 librados: watch_flush enter 2024-03-29T11:24:01.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 10 librados: watch_flush exit 2024-03-29T11:24:01.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:24:01.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 10 monclient: shutdown 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 >> v1:172.21.15.136:6789/0 conn(0x7faab009ac40 legacy=0x7faab01dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 shutdown_connections 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 >> 172.21.15.136:0/1206935015 conn(0x7faab0066b00 msgr2=0x7faab013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 shutdown_connections 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 1 -- 172.21.15.136:0/1206935015 wait complete. 2024-03-29T11:24:01.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.175+0000 7faab5b32640 1 librados: shutdown 2024-03-29T11:24:01.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 6d0eda31-ed06-482c-8ff6-a9ed9f1f148b 4 2024-03-29T11:24:01.201 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.220+0000 7fdf4d3f1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:24:01.205 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.224+0000 7f4941e2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:24:01.274 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.293+0000 7fdf4d3f1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:01.278 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.298+0000 7f4941e2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:01.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.323+0000 7f5b8473d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.323+0000 7f5b8473d640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.323+0000 7f5b8473d640 10 monclient: monmap: 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:01.324499+0000 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:01.324499+0000 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:01.304 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 1 Processor -- start 2024-03-29T11:24:01.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 1 -- start start 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient: init 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b7c09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b7c09c870 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7c0b4360 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c09dec0 con 0x7f5b7c09ac40 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c09da30 con 0x7f5b7c09c870 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c0b39c0 con 0x7f5b7c0b4360 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.324+0000 7f5b8473d640 10 monclient(hunting): authenticate will time out at 775.496755s 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b81cd8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5b7c09c870 0x7f5b7c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39848/0 (socket says 172.21.15.136:39848) 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b81cd8640 1 -- 172.21.15.136:0/190700191 learned_addr learned my addr 172.21.15.136:0/190700191 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752646218 0 0) 0x7f5b7c09da30 con 0x7f5b7c09c870 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b68003520 con 0x7f5b7c09c870 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315187502 0 0) 0x7f5b7c0b39c0 con 0x7f5b7c0b4360 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c09da30 con 0x7f5b7c0b4360 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304849658 0 0) 0x7f5b78001110 con 0x7f5b7c09ac40 2024-03-29T11:24:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c0b39c0 con 0x7f5b7c09ac40 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1990602266 0 0) 0x7f5b68003520 con 0x7f5b7c09c870 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 >> v1:172.21.15.64:6790/0 conn(0x7f5b7c0b4360 legacy=0x7f5b7c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 >> v1:172.21.15.64:6789/0 conn(0x7f5b7c09ac40 legacy=0x7f5b7c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 monclient: found mon.noname-c 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0b5160 con 0x7f5b7c09c870 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5b6c001690 con 0x7f5b7c09c870 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: dump: 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b8473d640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b8473d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b6c001870 con 0x7f5b7c09c870 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 1 -- 172.21.15.136:0/190700191 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5b6c001c80 con 0x7f5b7c09c870 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b814d7640 10 monclient: dump: 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.313 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b8473d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.325+0000 7f5b8473d640 4 set_mon_vals no callback set 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 10 monclient: shutdown 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 1 -- 172.21.15.136:0/190700191 >> v1:172.21.15.136:6789/0 conn(0x7f5b7c09c870 legacy=0x7f5b7c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 1 -- 172.21.15.136:0/190700191 shutdown_connections 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 1 -- 172.21.15.136:0/190700191 >> 172.21.15.136:0/190700191 conn(0x7f5b7c066b00 msgr2=0x7f5b7c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 1 -- 172.21.15.136:0/190700191 shutdown_connections 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.326+0000 7f5b8473d640 1 -- 172.21.15.136:0/190700191 wait complete. 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.327+0000 7f5b8473d640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.327+0000 7f5b8473d640 10 monclient: monmap: 2024-03-29T11:24:01.314 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.327+0000 7f5b8473d640 1 librados: starting msgr at 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.327+0000 7f5b8473d640 1 librados: starting objecter 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.328+0000 7f5b8473d640 1 Processor -- start 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.328+0000 7f5b8473d640 1 -- start start 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.328+0000 7f5b8473d640 1 librados: setting wanted keys 2024-03-29T11:24:01.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.328+0000 7f5b8473d640 1 librados: calling monclient init 2024-03-29T11:24:01.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.328+0000 7f5b8473d640 10 monclient: init 2024-03-29T11:24:01.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:01.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b7c09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b7c0b07a0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7c1d7cd0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c0acd70 con 0x7f5b7c09ac40 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c1d8100 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c11f8d0 con 0x7f5b7c0b07a0 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient(hunting): authenticate will time out at 775.500967s 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b82cda640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5b7c1d7cd0 0x7f5b7c1e1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39852/0 (socket says 172.21.15.136:39852) 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b82cda640 1 -- 172.21.15.136:0/1614904980 learned_addr learned my addr 172.21.15.136:0/1614904980 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 client.?.objecter ms_handle_connect 0x7f5b7c1d7cd0 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 client.?.objecter ms_handle_connect 0x7f5b7c09ac40 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 client.?.objecter ms_handle_connect 0x7f5b7c0b07a0 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268673324 0 0) 0x7f5b7c1d8100 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b60004eb0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775503453 0 0) 0x7f5b7c0acd70 con 0x7f5b7c09ac40 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c1d8100 con 0x7f5b7c09ac40 2024-03-29T11:24:01.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913580306 0 0) 0x7f5b7c11f8d0 con 0x7f5b7c0b07a0 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c0acd70 con 0x7f5b7c0b07a0 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1077196935 0 0) 0x7f5b60004eb0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b7c11f8d0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5b74001c00 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.323 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917560248 0 0) 0x7f5b7c0acd70 con 0x7f5b7c0b07a0 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b60004eb0 con 0x7f5b7c0b07a0 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5b6c001b80 con 0x7f5b7c0b07a0 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.324 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578653317 0 0) 0x7f5b7c1d8100 con 0x7f5b7c09ac40 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b7c0acd70 con 0x7f5b7c09ac40 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5b78001860 con 0x7f5b7c09ac40 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:01.325 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988450649 0 0) 0x7f5b7c11f8d0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 >> v1:172.21.15.64:6790/0 conn(0x7f5b7c0b07a0 legacy=0x7f5b7c1d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 >> v1:172.21.15.64:6789/0 conn(0x7f5b7c09ac40 legacy=0x7f5b7c0afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 monclient: found mon.b 2024-03-29T11:24:01.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c1e64c0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient: _renew_subs 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c1e3490 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient: _renew_subs 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.329+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c1e39c0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b8473d640 1 librados: init done 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b740020d0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5b74003b30 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b80cd6640 4 set_mon_vals no callback set 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 10 monclient: dump: 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.328 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5b740040e0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 12151202 0 0) 0x7f5b74005d30 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 10 client.4141.objecter ms_dispatch 0x7f5b7c000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b72ffd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b8473d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b8473d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b8473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.330+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64005080 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.332+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5b74004390 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.332+0000 7f5b72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.332+0000 7f5b72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.330 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.341+0000 7fdf4d3f1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:24:01.330 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.348+0000 7f4941e2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:24:01.376 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.396+0000 7fdf4d3f1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:24:01.381 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.400+0000 7f4941e2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:24:01.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.403+0000 7f5b8473d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d0eda31-ed06-482c-8ff6-a9ed9f1f148b", "id": 4}] 2024-03-29T11:24:01.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.403+0000 7f5b8473d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d0eda31-ed06-482c-8ff6-a9ed9f1f148b", "id": 4}] 2024-03-29T11:24:01.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.403+0000 7f5b8473d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.403+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6d0eda31-ed06-482c-8ff6-a9ed9f1f148b", "id": 4} v 0) v1 -- 0x7f5b640058b0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.411+0000 7f5b72ffd640 1 -- 172.21.15.136:0/1614904980 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d0eda31-ed06-482c-8ff6-a9ed9f1f148b", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3968386737 0 1697012583) 0x7f5b740045b0 con 0x7f5b7c1d7cd0 2024-03-29T11:24:01.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.411+0000 7f5b72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d0eda31-ed06-482c-8ff6-a9ed9f1f148b", "id": 4}] 2024-03-29T11:24:01.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.411+0000 7f5b72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.392 INFO:teuthology.orchestra.run.smithi136.stdout:4 2024-03-29T11:24:01.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 10 librados: watch_flush enter 2024-03-29T11:24:01.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 10 librados: watch_flush exit 2024-03-29T11:24:01.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:24:01.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 10 monclient: shutdown 2024-03-29T11:24:01.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 >> v1:172.21.15.136:6789/0 conn(0x7f5b7c1d7cd0 legacy=0x7f5b7c1e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 shutdown_connections 2024-03-29T11:24:01.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 >> 172.21.15.136:0/1614904980 conn(0x7f5b7c066b00 msgr2=0x7f5b7c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 shutdown_connections 2024-03-29T11:24:01.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 1 -- 172.21.15.136:0/1614904980 wait complete. 2024-03-29T11:24:01.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.412+0000 7f5b8473d640 1 librados: shutdown 2024-03-29T11:24:01.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 43b962f4-e6cb-4a70-a62f-13ac7700c10d 5 2024-03-29T11:24:01.426 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.445+0000 7fdf4d3f1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:24:01.432 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.452+0000 7f4941e2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:24:01.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.522+0000 7f3c868a8640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.522+0000 7f3c868a8640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.522+0000 7f3c868a8640 10 monclient: monmap: 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:01.523255+0000 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:01.523255+0000 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:01.503 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 1 Processor -- start 2024-03-29T11:24:01.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 1 -- start start 2024-03-29T11:24:01.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient: init 2024-03-29T11:24:01.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient(hunting): picked mon.noname-a con 0x7f3c800b8240 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient(hunting): picked mon.noname-c con 0x7f3c800b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient(hunting): picked mon.noname-b con 0x7f3c8009c8c0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8009bd80 con 0x7f3c800b8240 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8009b960 con 0x7f3c800b1f60 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800b2ec0 con 0x7f3c8009c8c0 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c868a8640 10 monclient(hunting): authenticate will time out at 775.695513s 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c7f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3c800b1f60 0x7f3c8009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39864/0 (socket says 172.21.15.136:39864) 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.523+0000 7f3c7f7fe640 1 -- 172.21.15.136:0/2194034802 learned_addr learned my addr 172.21.15.136:0/2194034802 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934562866 0 0) 0x7f3c8009b960 con 0x7f3c800b1f60 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c68003520 con 0x7f3c800b1f60 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366391841 0 0) 0x7f3c8009bd80 con 0x7f3c800b8240 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c8009b960 con 0x7f3c800b8240 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448153018 0 0) 0x7f3c800b2ec0 con 0x7f3c8009c8c0 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:24:01.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c8009bd80 con 0x7f3c8009c8c0 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415070879 0 0) 0x7f3c68003520 con 0x7f3c800b1f60 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 >> v1:172.21.15.64:6790/0 conn(0x7f3c8009c8c0 legacy=0x7f3c8014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 >> v1:172.21.15.64:6789/0 conn(0x7f3c800b8240 legacy=0x7f3c8013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 monclient: found mon.noname-c 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8013fb60 con 0x7f3c800b1f60 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3c70001690 con 0x7f3c800b1f60 2024-03-29T11:24:01.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: dump: 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c70001870 con 0x7f3c800b1f60 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 1 -- 172.21.15.136:0/2194034802 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3c70001c80 con 0x7f3c800b1f60 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 4 set_mon_vals no callback set 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c7effd640 10 monclient: dump: 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.517 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 10 monclient: shutdown 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 1 -- 172.21.15.136:0/2194034802 >> v1:172.21.15.136:6789/0 conn(0x7f3c800b1f60 legacy=0x7f3c8009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 1 -- 172.21.15.136:0/2194034802 shutdown_connections 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 1 -- 172.21.15.136:0/2194034802 >> 172.21.15.136:0/2194034802 conn(0x7f3c80066b00 msgr2=0x7f3c80066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.524+0000 7f3c868a8640 1 -- 172.21.15.136:0/2194034802 shutdown_connections 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.525+0000 7f3c868a8640 1 -- 172.21.15.136:0/2194034802 wait complete. 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.525+0000 7f3c868a8640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.525+0000 7f3c868a8640 10 monclient: monmap: 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.525+0000 7f3c868a8640 1 librados: starting msgr at 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.525+0000 7f3c868a8640 1 librados: starting objecter 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.525+0000 7f3c868a8640 1 Processor -- start 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 1 -- start start 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 1 librados: setting wanted keys 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 1 librados: calling monclient init 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient: init 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient(hunting): picked mon.noname-b con 0x7f3c8009c8c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient(hunting): picked mon.noname-c con 0x7f3c800b8240 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient(hunting): picked mon.noname-a con 0x7f3c80132da0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c801d7350 con 0x7f3c80132da0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80133290 con 0x7f3c8009c8c0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80145090 con 0x7f3c800b8240 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c868a8640 10 monclient(hunting): authenticate will time out at 775.698526s 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3c8009c8c0 0x7f3c80132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39874/0 (socket says 172.21.15.136:39874) 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7ffff640 1 -- 172.21.15.136:0/1511453287 learned_addr learned my addr 172.21.15.136:0/1511453287 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 client.?.objecter ms_handle_connect 0x7f3c8009c8c0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542284550 0 0) 0x7f3c80133290 con 0x7f3c8009c8c0 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c004eb0 con 0x7f3c8009c8c0 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 client.?.objecter ms_handle_connect 0x7f3c800b8240 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.526+0000 7f3c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 client.?.objecter ms_handle_connect 0x7f3c80132da0 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990949970 0 0) 0x7f3c80145090 con 0x7f3c800b8240 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80133290 con 0x7f3c800b8240 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 174782932 0 0) 0x7f3c6c004eb0 con 0x7f3c8009c8c0 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c80145090 con 0x7f3c8009c8c0 2024-03-29T11:24:01.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3c74001a30 con 0x7f3c8009c8c0 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098905960 0 0) 0x7f3c801d7350 con 0x7f3c80132da0 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c004eb0 con 0x7f3c80132da0 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1297835235 0 0) 0x7f3c80133290 con 0x7f3c800b8240 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c801d7350 con 0x7f3c800b8240 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3c70001b80 con 0x7f3c800b8240 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1468799218 0 0) 0x7f3c80145090 con 0x7f3c8009c8c0 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 >> v1:172.21.15.64:6790/0 conn(0x7f3c800b8240 legacy=0x7f3c80134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 >> v1:172.21.15.64:6789/0 conn(0x7f3c80132da0 legacy=0x7f3c801d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 monclient: found mon.b 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80145220 con 0x7f3c8009c8c0 2024-03-29T11:24:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 10 monclient: _renew_subs 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c80135420 con 0x7f3c8009c8c0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 10 monclient: _renew_subs 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c801d7e20 con 0x7f3c8009c8c0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c868a8640 1 librados: init done 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c74001d10 con 0x7f3c8009c8c0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3c74003a60 con 0x7f3c8009c8c0 2024-03-29T11:24:01.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 monclient: dump: 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7e7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3c74004010 con 0x7f3c8009c8c0 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3496352450 0 0) 0x7f3c740068c0 con 0x7f3c8009c8c0 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 10 client.4147.objecter ms_dispatch 0x7f3c80000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.527+0000 7f3c7cff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.528+0000 7f3c868a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.528+0000 7f3c868a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.528+0000 7f3c868a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.528+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60005080 con 0x7f3c8009c8c0 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.530+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3c74006b20 con 0x7f3c8009c8c0 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.530+0000 7f3c7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.530+0000 7f3c7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.603+0000 7f3c868a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "43b962f4-e6cb-4a70-a62f-13ac7700c10d", "id": 5}] 2024-03-29T11:24:01.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.603+0000 7f3c868a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "43b962f4-e6cb-4a70-a62f-13ac7700c10d", "id": 5}] 2024-03-29T11:24:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.603+0000 7f3c868a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.603+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "43b962f4-e6cb-4a70-a62f-13ac7700c10d", "id": 5} v 0) v1 -- 0x7f3c600058b0 con 0x7f3c8009c8c0 2024-03-29T11:24:01.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.610+0000 7f3c7cff9640 1 -- 172.21.15.136:0/1511453287 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "43b962f4-e6cb-4a70-a62f-13ac7700c10d", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1853725142 0 1988412176) 0x7f3c74006340 con 0x7f3c8009c8c0 2024-03-29T11:24:01.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "43b962f4-e6cb-4a70-a62f-13ac7700c10d", "id": 5}] 2024-03-29T11:24:01.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.591 INFO:teuthology.orchestra.run.smithi136.stdout:5 2024-03-29T11:24:01.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 10 librados: watch_flush enter 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 10 librados: watch_flush exit 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 10 monclient: shutdown 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 >> v1:172.21.15.136:6789/0 conn(0x7f3c8009c8c0 legacy=0x7f3c80132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 shutdown_connections 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.611+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 >> 172.21.15.136:0/1511453287 conn(0x7f3c80066b00 msgr2=0x7f3c8013dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.612+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 shutdown_connections 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.612+0000 7f3c868a8640 1 -- 172.21.15.136:0/1511453287 wait complete. 2024-03-29T11:24:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.612+0000 7f3c868a8640 1 librados: shutdown 2024-03-29T11:24:01.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new aba450cc-a0a8-4404-95c9-a4b048cb684f 6 2024-03-29T11:24:01.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.762+0000 7f9b77e5c640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.762+0000 7f9b77e5c640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.762+0000 7f9b77e5c640 10 monclient: monmap: 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:01.763721+0000 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:01.763721+0000 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.763+0000 7f9b77e5c640 1 Processor -- start 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.763+0000 7f9b77e5c640 1 -- start start 2024-03-29T11:24:01.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.763+0000 7f9b77e5c640 10 monclient: init 2024-03-29T11:24:01.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f9b700b8240 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9b700b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7009c8c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7009bd80 con 0x7f9b700b8240 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7009b960 con 0x7f9b7009c8c0 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b700b2ec0 con 0x7f9b700b1f60 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient(hunting): authenticate will time out at 775.936002s 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b763f9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b7009c8c0 0x7f9b7014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39884/0 (socket says 172.21.15.136:39884) 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b763f9640 1 -- 172.21.15.136:0/3024531533 learned_addr learned my addr 172.21.15.136:0/3024531533 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928816178 0 0) 0x7f9b7009b960 con 0x7f9b7009c8c0 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c003520 con 0x7f9b7009c8c0 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044065054 0 0) 0x7f9b7009bd80 con 0x7f9b700b8240 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7009b960 con 0x7f9b700b8240 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042219083 0 0) 0x7f9b700b2ec0 con 0x7f9b700b1f60 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:24:01.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7009bd80 con 0x7f9b700b1f60 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823438812 0 0) 0x7f9b5c003520 con 0x7f9b7009c8c0 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 >> v1:172.21.15.64:6790/0 conn(0x7f9b700b1f60 legacy=0x7f9b7009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 >> v1:172.21.15.64:6789/0 conn(0x7f9b700b8240 legacy=0x7f9b7013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 monclient: found mon.noname-c 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7013fb60 con 0x7f9b7009c8c0 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9b68001a10 con 0x7f9b7009c8c0 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: dump: 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b68001be0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 1 -- 172.21.15.136:0/3024531533 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9b68001fb0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b77e5c640 4 set_mon_vals no callback set 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.764+0000 7f9b74bf6640 10 monclient: dump: 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 10 monclient: shutdown 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 1 -- 172.21.15.136:0/3024531533 >> v1:172.21.15.136:6789/0 conn(0x7f9b7009c8c0 legacy=0x7f9b7014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 1 -- 172.21.15.136:0/3024531533 shutdown_connections 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 1 -- 172.21.15.136:0/3024531533 >> 172.21.15.136:0/3024531533 conn(0x7f9b70066b00 msgr2=0x7f9b70066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 1 -- 172.21.15.136:0/3024531533 shutdown_connections 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.765+0000 7f9b77e5c640 1 -- 172.21.15.136:0/3024531533 wait complete. 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 10 monclient: monmap: 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 1 librados: starting msgr at 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 1 librados: starting objecter 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 1 Processor -- start 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 1 -- start start 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 1 librados: setting wanted keys 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 1 librados: calling monclient init 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.766+0000 7f9b77e5c640 10 monclient: init 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7009c8c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f9b700b8240 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f9b701327f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b70132f10 con 0x7f9b700b8240 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b70133d60 con 0x7f9b7009c8c0 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b70134040 con 0x7f9b701327f0 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient(hunting): authenticate will time out at 775.938939s 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b75bf8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b7009c8c0 0x7f9b70144b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39892/0 (socket says 172.21.15.136:39892) 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b75bf8640 1 -- 172.21.15.136:0/2608520056 learned_addr learned my addr 172.21.15.136:0/2608520056 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 client.?.objecter ms_handle_connect 0x7f9b7009c8c0 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 client.?.objecter ms_handle_connect 0x7f9b700b8240 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 client.?.objecter ms_handle_connect 0x7f9b701327f0 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157391347 0 0) 0x7f9b70133d60 con 0x7f9b7009c8c0 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54004eb0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239912229 0 0) 0x7f9b70132f10 con 0x7f9b700b8240 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b70133d60 con 0x7f9b700b8240 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558929998 0 0) 0x7f9b70134040 con 0x7f9b701327f0 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b70132f10 con 0x7f9b701327f0 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1106324688 0 0) 0x7f9b54004eb0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b70134040 con 0x7f9b7009c8c0 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9b6c001940 con 0x7f9b7009c8c0 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): dump: 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.763 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158654608 0 0) 0x7f9b70133d60 con 0x7f9b700b8240 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b54004eb0 con 0x7f9b700b8240 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9b60001870 con 0x7f9b700b8240 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): dump: 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.764 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179537553 0 0) 0x7f9b70132f10 con 0x7f9b701327f0 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b70133d60 con 0x7f9b701327f0 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9b68001d50 con 0x7f9b701327f0 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient(hunting): dump: 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3746973027 0 0) 0x7f9b70134040 con 0x7f9b7009c8c0 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 >> v1:172.21.15.64:6790/0 conn(0x7f9b701327f0 legacy=0x7f9b701e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 >> v1:172.21.15.64:6789/0 conn(0x7f9b700b8240 legacy=0x7f9b701452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 monclient: found mon.b 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b70134320 con 0x7f9b7009c8c0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient: _renew_subs 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b70131c50 con 0x7f9b7009c8c0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient: _renew_subs 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b701e3a10 con 0x7f9b7009c8c0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b77e5c640 1 librados: init done 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.767+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b6c0035f0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9b6c003980 con 0x7f9b7009c8c0 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b67fff640 4 set_mon_vals no callback set 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 10 monclient: dump: 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b6c003e70 con 0x7f9b7009c8c0 2024-03-29T11:24:01.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 101318270 0 0) 0x7f9b6c0067c0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 10 client.4153.objecter ms_dispatch 0x7f9b70000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b667fc640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b77e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b77e5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b77e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.768+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b58005080 con 0x7f9b7009c8c0 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.770+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b6c004410 con 0x7f9b7009c8c0 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.770+0000 7f9b667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.770+0000 7f9b667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.842+0000 7f9b77e5c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aba450cc-a0a8-4404-95c9-a4b048cb684f", "id": 6}] 2024-03-29T11:24:01.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.842+0000 7f9b77e5c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aba450cc-a0a8-4404-95c9-a4b048cb684f", "id": 6}] 2024-03-29T11:24:01.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.842+0000 7f9b77e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.842+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aba450cc-a0a8-4404-95c9-a4b048cb684f", "id": 6} v 0) v1 -- 0x7f9b580058b0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.826 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.846+0000 7fdf4d3f1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:24:01.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b667fc640 1 -- 172.21.15.136:0/2608520056 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aba450cc-a0a8-4404-95c9-a4b048cb684f", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1021751550 0 1113811849) 0x7f9b6c0062a0 con 0x7f9b7009c8c0 2024-03-29T11:24:01.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aba450cc-a0a8-4404-95c9-a4b048cb684f", "id": 6}] 2024-03-29T11:24:01.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.828 INFO:teuthology.orchestra.run.smithi136.stdout:6 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b77e5c640 10 librados: watch_flush enter 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b77e5c640 10 librados: watch_flush exit 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b77e5c640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.848+0000 7f9b77e5c640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 10 monclient: shutdown 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 >> v1:172.21.15.136:6789/0 conn(0x7f9b7009c8c0 legacy=0x7f9b70144b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 shutdown_connections 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 >> 172.21.15.136:0/2608520056 conn(0x7f9b70066b00 msgr2=0x7f9b70069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 shutdown_connections 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 1 -- 172.21.15.136:0/2608520056 wait complete. 2024-03-29T11:24:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.849+0000 7f9b77e5c640 1 librados: shutdown 2024-03-29T11:24:01.833 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.853+0000 7f4941e2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:24:01.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 406b222b-5971-4483-bc6a-88ac425573d2 7 2024-03-29T11:24:01.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.959+0000 7fb863d0e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.959+0000 7fb863d0e640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.959+0000 7fb863d0e640 10 monclient: monmap: 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:24:01.960010+0000 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:24:01.960010+0000 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.959+0000 7fb863d0e640 1 Processor -- start 2024-03-29T11:24:01.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.959+0000 7fb863d0e640 1 -- start start 2024-03-29T11:24:01.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.959+0000 7fb863d0e640 10 monclient: init 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb85c09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb85c09c870 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb85c0b4360 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c09dec0 con 0x7fb85c09ac40 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c09da30 con 0x7fb85c09c870 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c0b39c0 con 0x7fb85c0b4360 2024-03-29T11:24:01.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 10 monclient(hunting): authenticate will time out at 776.132214s 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb8612a9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb85c09c870 0x7fb85c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39906/0 (socket says 172.21.15.136:39906) 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb8612a9640 1 -- 172.21.15.136:0/3207416218 learned_addr learned my addr 172.21.15.136:0/3207416218 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606532824 0 0) 0x7fb85c09da30 con 0x7fb85c09c870 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb848003520 con 0x7fb85c09c870 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610903910 0 0) 0x7fb84c001110 con 0x7fb85c09ac40 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c09da30 con 0x7fb85c09ac40 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296080652 0 0) 0x7fb85c0b39c0 con 0x7fb85c0b4360 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c001110 con 0x7fb85c0b4360 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3917740308 0 0) 0x7fb848003520 con 0x7fb85c09c870 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 >> v1:172.21.15.64:6790/0 conn(0x7fb85c0b4360 legacy=0x7fb85c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 >> v1:172.21.15.64:6789/0 conn(0x7fb85c09ac40 legacy=0x7fb85c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 monclient: found mon.noname-c 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c0b5160 con 0x7fb85c09c870 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb858001690 con 0x7fb85c09c870 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb863d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.960+0000 7fb860aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 10 monclient: dump: 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.952 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb858001870 con 0x7fb85c09c870 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 1 -- 172.21.15.136:0/3207416218 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb858001c80 con 0x7fb85c09c870 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 4 set_mon_vals no callback set 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb860aa8640 10 monclient: dump: 2024-03-29T11:24:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 10 monclient: shutdown 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 1 -- 172.21.15.136:0/3207416218 >> v1:172.21.15.136:6789/0 conn(0x7fb85c09c870 legacy=0x7fb85c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 1 -- 172.21.15.136:0/3207416218 shutdown_connections 2024-03-29T11:24:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 1 -- 172.21.15.136:0/3207416218 >> 172.21.15.136:0/3207416218 conn(0x7fb85c066b00 msgr2=0x7fb85c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 1 -- 172.21.15.136:0/3207416218 shutdown_connections 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.961+0000 7fb863d0e640 1 -- 172.21.15.136:0/3207416218 wait complete. 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.962+0000 7fb863d0e640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.962+0000 7fb863d0e640 10 monclient: monmap: 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 1 librados: starting msgr at 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 1 librados: starting objecter 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 1 Processor -- start 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 1 -- start start 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 1 librados: setting wanted keys 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 1 librados: calling monclient init 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.963+0000 7fb863d0e640 10 monclient: init 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb85c09ac40 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb85c0b7eb0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb85c13ffa0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c143060 con 0x7fb85c13ffa0 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c1d7760 con 0x7fb85c09ac40 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c1d78f0 con 0x7fb85c0b7eb0 2024-03-29T11:24:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient(hunting): authenticate will time out at 776.136096s 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb861aaa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb85c09ac40 0x7fb85c1427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39912/0 (socket says 172.21.15.136:39912) 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb861aaa640 1 -- 172.21.15.136:0/3217455536 learned_addr learned my addr 172.21.15.136:0/3217455536 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 client.?.objecter ms_handle_connect 0x7fb85c09ac40 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 client.?.objecter ms_handle_connect 0x7fb85c0b7eb0 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839124172 0 0) 0x7fb85c1d7760 con 0x7fb85c09ac40 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb840004eb0 con 0x7fb85c09ac40 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 client.?.objecter ms_handle_connect 0x7fb85c13ffa0 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129974873 0 0) 0x7fb85c1d78f0 con 0x7fb85c0b7eb0 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c1d7760 con 0x7fb85c0b7eb0 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918337632 0 0) 0x7fb85c143060 con 0x7fb85c13ffa0 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c1d78f0 con 0x7fb85c13ffa0 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3992955716 0 0) 0x7fb840004eb0 con 0x7fb85c09ac40 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb85c143060 con 0x7fb85c09ac40 2024-03-29T11:24:01.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb84c0019a0 con 0x7fb85c09ac40 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): dump: 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1741629827 0 0) 0x7fb85c1d7760 con 0x7fb85c0b7eb0 2024-03-29T11:24:01.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb840004eb0 con 0x7fb85c0b7eb0 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb858001b80 con 0x7fb85c0b7eb0 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient(hunting): dump: 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1862330083 0 0) 0x7fb85c143060 con 0x7fb85c09ac40 2024-03-29T11:24:01.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 >> v1:172.21.15.64:6790/0 conn(0x7fb85c0b7eb0 legacy=0x7fb85c1d7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 >> v1:172.21.15.64:6789/0 conn(0x7fb85c13ffa0 legacy=0x7fb85c1e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 monclient: found mon.b 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c140460 con 0x7fb85c09ac40 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient: _renew_subs 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c140710 con 0x7fb85c09ac40 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient: _renew_subs 2024-03-29T11:24:01.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.964+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c1e3390 con 0x7fb85c09ac40 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb863d0e640 1 librados: init done 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb84c003610 con 0x7fb85c09ac40 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb84c003920 con 0x7fb85c09ac40 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb853fff640 4 set_mon_vals no callback set 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 10 monclient: dump: 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:01.962 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb84c003ed0 con 0x7fb85c09ac40 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2260845790 0 0) 0x7fb84c0067e0 con 0x7fb85c09ac40 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 10 client.4159.objecter ms_dispatch 0x7fb85c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb8527fc640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb863d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb863d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb863d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.965+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb844005080 con 0x7fb85c09ac40 2024-03-29T11:24:01.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.967+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb84c004310 con 0x7fb85c09ac40 2024-03-29T11:24:01.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.967+0000 7fb8527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:01.967+0000 7fb8527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.968 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:01.987+0000 7fdf4d3f1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:24:01.977 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:01.997+0000 7f4941e2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:24:02.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.040+0000 7fb863d0e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "406b222b-5971-4483-bc6a-88ac425573d2", "id": 7}] 2024-03-29T11:24:02.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.040+0000 7fb863d0e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "406b222b-5971-4483-bc6a-88ac425573d2", "id": 7}] 2024-03-29T11:24:02.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.040+0000 7fb863d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:02.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.040+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "406b222b-5971-4483-bc6a-88ac425573d2", "id": 7} v 0) v1 -- 0x7fb8440058b0 con 0x7fb85c09ac40 2024-03-29T11:24:02.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.044+0000 7fb8527fc640 1 -- 172.21.15.136:0/3217455536 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "406b222b-5971-4483-bc6a-88ac425573d2", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4051149069 0 1371665406) 0x7fb84c006250 con 0x7fb85c09ac40 2024-03-29T11:24:02.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.044+0000 7fb8527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "406b222b-5971-4483-bc6a-88ac425573d2", "id": 7}] 2024-03-29T11:24:02.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.044+0000 7fb8527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:02.025 INFO:teuthology.orchestra.run.smithi136.stdout:7 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 10 librados: watch_flush enter 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 10 librados: watch_flush exit 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 10 client.4159.objecter successfully canceled tick 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 10 monclient: shutdown 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 >> v1:172.21.15.136:6789/0 conn(0x7fb85c09ac40 legacy=0x7fb85c1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 shutdown_connections 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 >> 172.21.15.136:0/3217455536 conn(0x7fb85c066b00 msgr2=0x7fb85c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 shutdown_connections 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 1 -- 172.21.15.136:0/3217455536 wait complete. 2024-03-29T11:24:02.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T11:24:02.045+0000 7fb863d0e640 1 librados: shutdown 2024-03-29T11:24:02.035 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:24:02.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:24:02.038 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:24:02.038 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:24:02.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:24:02.040 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:24:02.040 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:24:02.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:24:02.042 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:24:02.042 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:24:02.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:24:02.044 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:24:02.044 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:24:02.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:24:02.047 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:24:02.047 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:24:02.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:24:02.049 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:24:02.049 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:24:02.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:24:02.051 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:24:02.051 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:24:02.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:24:02.053 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:24:02.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:02.106 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.125+0000 7fdf4d3f1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:24:02.114 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.134+0000 7f4941e2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:24:02.167 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.187+0000 7f4941e2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:24:02.182 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.202+0000 7fdf4d3f1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:24:02.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.216+0000 7f1b4884e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:02.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.216+0000 7f1b4884e640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.216+0000 7f1b4884e640 10 monclient: monmap: 2024-03-29T11:24:02.196 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T11:24:02.196 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:24:02.217222+0000 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:24:02.217222+0000 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:02.197 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.216+0000 7f1b4884e640 1 Processor -- start 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.216+0000 7f1b4884e640 1 -- start start 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.216+0000 7f1b4884e640 10 monclient: init 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b4409ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b4409c870 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b440b4360 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4409dec0 con 0x7f1b440b4360 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4409da30 con 0x7f1b4409c870 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440b39c0 con 0x7f1b4409ac40 2024-03-29T11:24:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b4884e640 10 monclient(hunting): authenticate will time out at 1178.487580s 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b42d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1b440b4360 0x7f1b4414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55798/0 (socket says 172.21.15.64:55798) 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.217+0000 7f1b42d9d640 1 -- 172.21.15.64:0/818903051 learned_addr learned my addr 172.21.15.64:0/818903051 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520316110 0 0) 0x7f1b440b39c0 con 0x7f1b4409ac40 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c003520 con 0x7f1b4409ac40 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943841926 0 0) 0x7f1b3c001490 con 0x7f1b440b4360 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440b39c0 con 0x7f1b440b4360 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080680568 0 0) 0x7f1b4409da30 con 0x7f1b4409c870 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3c001490 con 0x7f1b4409c870 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909295084 0 0) 0x7f1b2c003520 con 0x7f1b4409ac40 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 >> v1:172.21.15.136:6789/0 conn(0x7f1b4409c870 legacy=0x7f1b4413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 >> v1:172.21.15.64:6789/0 conn(0x7f1b440b4360 legacy=0x7f1b4414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 monclient: found mon.noname-b 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440b5160 con 0x7f1b4409ac40 2024-03-29T11:24:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4884e640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4884e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b38001690 con 0x7f1b4409ac40 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: dump: 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4884e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b38001870 con 0x7f1b4409ac40 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 1 -- 172.21.15.64:0/818903051 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b38001cc0 con 0x7f1b4409ac40 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4159a640 10 monclient: dump: 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4884e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.218+0000 7f1b4884e640 4 set_mon_vals no callback set 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 10 monclient: shutdown 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 1 -- 172.21.15.64:0/818903051 >> v1:172.21.15.64:6790/0 conn(0x7f1b4409ac40 legacy=0x7f1b440b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 1 -- 172.21.15.64:0/818903051 shutdown_connections 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 1 -- 172.21.15.64:0/818903051 >> 172.21.15.64:0/818903051 conn(0x7f1b44066b00 msgr2=0x7f1b44066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 1 -- 172.21.15.64:0/818903051 shutdown_connections 2024-03-29T11:24:02.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.219+0000 7f1b4884e640 1 -- 172.21.15.64:0/818903051 wait complete. 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.220+0000 7f1b4884e640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.220+0000 7f1b4884e640 10 monclient: monmap: 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:02.207 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 1 librados: starting msgr at 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 1 librados: starting objecter 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 1 Processor -- start 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 1 -- start start 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 1 librados: setting wanted keys 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 1 librados: calling monclient init 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.221+0000 7f1b4884e640 10 monclient: init 2024-03-29T11:24:02.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b4409ac40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b440af6a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b441d7690 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b441da250 con 0x7f1b440af6a0 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b441da4f0 con 0x7f1b441d7690 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b441db100 con 0x7f1b4409ac40 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4884e640 10 monclient(hunting): authenticate will time out at 1178.492205s 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1b4409ac40 0x7f1b441da9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50220/0 (socket says 172.21.15.64:50220) 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b4259c640 1 -- 172.21.15.64:0/2132587189 learned_addr learned my addr 172.21.15.64:0/2132587189 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 client.?.objecter ms_handle_connect 0x7f1b4409ac40 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 client.?.objecter ms_handle_connect 0x7f1b440af6a0 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 client.?.objecter ms_handle_connect 0x7f1b441d7690 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718177500 0 0) 0x7f1b441db100 con 0x7f1b4409ac40 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24004eb0 con 0x7f1b4409ac40 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545425691 0 0) 0x7f1b441da4f0 con 0x7f1b441d7690 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b441db100 con 0x7f1b441d7690 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928153803 0 0) 0x7f1b441da250 con 0x7f1b440af6a0 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b441da4f0 con 0x7f1b440af6a0 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 337127306 0 0) 0x7f1b24004eb0 con 0x7f1b4409ac40 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b441da250 con 0x7f1b4409ac40 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b38001c80 con 0x7f1b4409ac40 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:24:02.215 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.222+0000 7f1b32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853799932 0 0) 0x7f1b441da4f0 con 0x7f1b440af6a0 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b24004eb0 con 0x7f1b440af6a0 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b34001830 con 0x7f1b440af6a0 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:02.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2987752064 0 0) 0x7f1b441da250 con 0x7f1b4409ac40 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 >> v1:172.21.15.136:6789/0 conn(0x7f1b441d7690 legacy=0x7f1b441e17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 >> v1:172.21.15.64:6789/0 conn(0x7f1b440af6a0 legacy=0x7f1b441d6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 monclient: found mon.c 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b441d7af0 con 0x7f1b4409ac40 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 10 monclient: _renew_subs 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b441e2f10 con 0x7f1b4409ac40 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 10 monclient: _renew_subs 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:02.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b441e3280 con 0x7f1b4409ac40 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b4884e640 1 librados: init done 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b38003960 con 0x7f1b4409ac40 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b38003c70 con 0x7f1b4409ac40 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 monclient: dump: 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b40d99640 4 set_mon_vals no callback set 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1b38004220 con 0x7f1b4409ac40 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f1b380064a0 con 0x7f1b4409ac40 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 10 client.4184.objecter ms_dispatch 0x7f1b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.223+0000 7f1b32ffd640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.224+0000 7f1b4884e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.224+0000 7f1b4884e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.224+0000 7f1b4884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.224+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b28005080 con 0x7f1b4409ac40 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.226+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1b380066b0 con 0x7f1b4409ac40 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.226+0000 7f1b32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.226+0000 7f1b32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:02.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.287+0000 7f1b4884e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.287+0000 7f1b4884e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.287+0000 7f1b4884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:02.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.287+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b28005330 con 0x7f1b4409ac40 2024-03-29T11:24:02.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.288+0000 7f1b32ffd640 1 -- 172.21.15.64:0/2132587189 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f1b380068d0 con 0x7f1b4409ac40 2024-03-29T11:24:02.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.288+0000 7f1b32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.288+0000 7f1b32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:02.269 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T11:24:02.269 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:02.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 10 librados: watch_flush enter 2024-03-29T11:24:02.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 10 librados: watch_flush exit 2024-03-29T11:24:02.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 10 monclient: shutdown 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 >> v1:172.21.15.64:6790/0 conn(0x7f1b4409ac40 legacy=0x7f1b441da9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 shutdown_connections 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 >> 172.21.15.64:0/2132587189 conn(0x7f1b44066b00 msgr2=0x7f1b4413a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 shutdown_connections 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 1 -- 172.21.15.64:0/2132587189 wait complete. 2024-03-29T11:24:02.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T11:24:02.289+0000 7f1b4884e640 1 librados: shutdown 2024-03-29T11:24:02.280 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:24:02.280 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:24:02.281 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.299+0000 7f4941e2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:02.303 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.323+0000 7fdf4d3f1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:02.374 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.394+0000 7f4941e2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:24:02.402 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.422+0000 7fdf4d3f1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:24:02.529 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.549+0000 7f4941e2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:02.564 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.583+0000 7fdf4d3f1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:02.571 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.590+0000 7f4941e2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:02.608 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.627+0000 7fdf4d3f1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:02.648 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-03-29T11:24:02.667+0000 7f7879815640 -1 Falling back to public interface 2024-03-29T11:24:02.648 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-03-29T11:24:02.668+0000 7f1d5f668640 -1 Falling back to public interface 2024-03-29T11:24:02.654 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-03-29T11:24:02.674+0000 7fdb7bf1f640 -1 Falling back to public interface 2024-03-29T11:24:02.657 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-03-29T11:24:02.676+0000 7f5bf21fa640 -1 Falling back to public interface 2024-03-29T11:24:02.661 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-29T11:24:02.680+0000 7fcad82d9640 -1 Falling back to public interface 2024-03-29T11:24:02.663 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-29T11:24:02.683+0000 7f960280e640 -1 Falling back to public interface 2024-03-29T11:24:02.663 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-03-29T11:24:02.683+0000 7f46dc39a640 -1 Falling back to public interface 2024-03-29T11:24:02.664 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-29T11:24:02.683+0000 7f96011ec640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T11:24:02.669 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-03-29T11:24:02.688+0000 7f85526bd640 -1 Falling back to public interface 2024-03-29T11:24:02.728 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:02.748+0000 7f4941e2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:24:02.773 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:02.792+0000 7fdf4d3f1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:24:03.085 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:03.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.195+0000 7f27c3033640 10 monclient: get_monmap_and_config 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.195+0000 7f27c3033640 10 monclient: build_initial_monmap 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.195+0000 7f27c3033640 10 monclient: monmap: 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:03.196234+0000 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:03.196234+0000 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:03.176 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 1 Processor -- start 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 1 -- start start 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient: init 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc1422b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc142690 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient(hunting): picked mon.noname-b con 0x7f27bc0b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc142c80 con 0x7f27bc1422b0 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0aae40 con 0x7f27bc142690 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc13fbd0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c3033640 10 monclient(hunting): authenticate will time out at 1179.466609s 2024-03-29T11:24:03.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c15d0640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f27bc0b17d0 0x7f27bc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50226/0 (socket says 172.21.15.64:50226) 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.196+0000 7f27c15d0640 1 -- 172.21.15.64:0/2161695144 learned_addr learned my addr 172.21.15.64:0/2161695144 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101226808 0 0) 0x7f27bc13fbd0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ac003520 con 0x7f27bc0b17d0 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687632474 0 0) 0x7f27bc142c80 con 0x7f27bc1422b0 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc13fbd0 con 0x7f27bc1422b0 2024-03-29T11:24:03.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633764814 0 0) 0x7f27bc0aae40 con 0x7f27bc142690 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc142c80 con 0x7f27bc142690 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2214491096 0 0) 0x7f27ac003520 con 0x7f27bc0b17d0 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 >> v1:172.21.15.136:6789/0 conn(0x7f27bc142690 legacy=0x7f27bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 >> v1:172.21.15.64:6789/0 conn(0x7f27bc1422b0 legacy=0x7f27bc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc151940 con 0x7f27bc0b17d0 2024-03-29T11:24:03.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f27b4001a10 con 0x7f27bc0b17d0 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27c3033640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27c3033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: dump: 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.188 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27c3033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27b4001bf0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 1 -- 172.21.15.64:0/2161695144 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f27b4001f80 con 0x7f27bc0b17d0 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27bb7fe640 10 monclient: dump: 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.189 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27c3033640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.197+0000 7f27c3033640 4 set_mon_vals no callback set 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 10 monclient: shutdown 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 1 -- 172.21.15.64:0/2161695144 >> v1:172.21.15.64:6790/0 conn(0x7f27bc0b17d0 legacy=0x7f27bc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 1 -- 172.21.15.64:0/2161695144 shutdown_connections 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 1 -- 172.21.15.64:0/2161695144 >> 172.21.15.64:0/2161695144 conn(0x7f27bc066a00 msgr2=0x7f27bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:03.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 1 -- 172.21.15.64:0/2161695144 shutdown_connections 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 1 -- 172.21.15.64:0/2161695144 wait complete. 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 10 monclient: build_initial_monmap 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.198+0000 7f27c3033640 10 monclient: monmap: 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 1 librados: starting msgr at 2024-03-29T11:24:03.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 1 librados: starting objecter 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 1 Processor -- start 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 1 -- start start 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 1 librados: setting wanted keys 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 1 librados: calling monclient init 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 10 monclient: init 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 10 monclient(hunting): picked mon.noname-b con 0x7f27bc0b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc1422b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.199+0000 7f27c3033640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc142690 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0aebd0 con 0x7f27bc1422b0 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0b10d0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0af450 con 0x7f27bc142690 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 10 monclient(hunting): authenticate will time out at 1179.469909s 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27bbfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f27bc1422b0 0x7f27bc0af8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55822/0 (socket says 172.21.15.64:55822) 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27bbfff640 1 -- 172.21.15.64:0/410350284 learned_addr learned my addr 172.21.15.64:0/410350284 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 client.?.objecter ms_handle_connect 0x7f27bc1422b0 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 client.?.objecter ms_handle_connect 0x7f27bc0b17d0 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 client.?.objecter ms_handle_connect 0x7f27bc142690 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761889797 0 0) 0x7f27bc0aebd0 con 0x7f27bc1422b0 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0004eb0 con 0x7f27bc1422b0 2024-03-29T11:24:03.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758409677 0 0) 0x7f27bc0b10d0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc0aebd0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274891287 0 0) 0x7f27bc0af450 con 0x7f27bc142690 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc0b10d0 con 0x7f27bc142690 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636834726 0 0) 0x7f27bc0aebd0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27bc0af450 con 0x7f27bc0b17d0 2024-03-29T11:24:03.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f27b00018a0 con 0x7f27bc0b17d0 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2622342205 0 0) 0x7f27a0004eb0 con 0x7f27bc1422b0 2024-03-29T11:24:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27bc0aebd0 con 0x7f27bc1422b0 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f27a8001970 con 0x7f27bc1422b0 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610617548 0 0) 0x7f27bc0b10d0 con 0x7f27bc142690 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27a0004eb0 con 0x7f27bc142690 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f27b4001d60 con 0x7f27bc142690 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 750630478 0 0) 0x7f27bc0aebd0 con 0x7f27bc1422b0 2024-03-29T11:24:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 >> v1:172.21.15.64:6790/0 conn(0x7f27bc142690 legacy=0x7f27bc1e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 >> v1:172.21.15.136:6789/0 conn(0x7f27bc0b17d0 legacy=0x7f27bc14dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 monclient: found mon.a 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0b1320 con 0x7f27bc1422b0 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 10 monclient: _renew_subs 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.200+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc13c430 con 0x7f27bc1422b0 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:03.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 10 monclient: _renew_subs 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc0af110 con 0x7f27bc1422b0 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 1 librados: init done 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a8003610 con 0x7f27bc1422b0 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f27a8003a20 con 0x7f27bc1422b0 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27baffd640 4 set_mon_vals no callback set 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 10 monclient: dump: 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.199 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f27a8003f90 con 0x7f27bc1422b0 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f27a8006940 con 0x7f27bc1422b0 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 10 client.4191.objecter ms_dispatch 0x7f27bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27b97fa640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.201+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279c005080 con 0x7f27bc1422b0 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.204+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f27a8004420 con 0x7f27bc1422b0 2024-03-29T11:24:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.204+0000 7f27b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:03.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.204+0000 7f27b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:03.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.266+0000 7f27c3033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:03.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.266+0000 7f27c3033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:03.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.266+0000 7f27c3033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:03.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.266+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f279c005330 con 0x7f27bc1422b0 2024-03-29T11:24:03.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27b97fa640 1 -- 172.21.15.64:0/410350284 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f27a8006320 con 0x7f27bc1422b0 2024-03-29T11:24:03.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:03.248 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:03.248 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:03.248 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27c3033640 10 librados: watch_flush enter 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27c3033640 10 librados: watch_flush exit 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27c3033640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.267+0000 7f27c3033640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 10 monclient: shutdown 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 >> v1:172.21.15.64:6789/0 conn(0x7f27bc1422b0 legacy=0x7f27bc0af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 shutdown_connections 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 >> 172.21.15.64:0/410350284 conn(0x7f27bc066a00 msgr2=0x7f27bc13d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:03.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 shutdown_connections 2024-03-29T11:24:03.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 1 -- 172.21.15.64:0/410350284 wait complete. 2024-03-29T11:24:03.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:03.268+0000 7f27c3033640 1 librados: shutdown 2024-03-29T11:24:03.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:03.531 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:03.550+0000 7f4941e2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:24:03.611 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:03.630+0000 7fdf4d3f1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:24:03.803 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:03.822+0000 7f4941e2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:24:03.854 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:03.874+0000 7f4941e2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:24:03.897 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:03.916+0000 7fdf4d3f1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:24:03.950 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:03.970+0000 7fdf4d3f1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:24:03.967 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:03.987+0000 7f4941e2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:24:04.076 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T11:24:04.095+0000 7fdf4d3f1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:24:04.986 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:05.005+0000 7f48e227d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:24:04.986 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T11:24:05.005+0000 7f48e227d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:24:06.119 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-03-29T11:24:06.139+0000 7f7879815640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:24:06.167 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-29T11:24:06.187+0000 7fcad82d9640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:24:06.198 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-03-29T11:24:06.218+0000 7f5bf21fa640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:24:06.207 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-03-29T11:24:06.226+0000 7f1d5f668640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:24:06.212 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-29T11:24:06.232+0000 7f960280e640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:24:06.214 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-03-29T11:24:06.233+0000 7f46dc39a640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:24:06.217 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-03-29T11:24:06.237+0000 7fdb7bf1f640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:24:06.232 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-03-29T11:24:06.251+0000 7f85526bd640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:24:10.063 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:10.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.231+0000 7f8c3faa5640 10 monclient: get_monmap_and_config 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.231+0000 7f8c3faa5640 10 monclient: build_initial_monmap 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.231+0000 7f8c3faa5640 10 monclient: monmap: 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:10.232840+0000 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:10.232840+0000 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:10.213 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:10.214 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.232+0000 7f8c3faa5640 1 Processor -- start 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.233+0000 7f8c3faa5640 1 -- start start 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.233+0000 7f8c3faa5640 10 monclient: init 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.233+0000 7f8c3faa5640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 10 monclient(hunting): picked mon.noname-b con 0x7f8c381422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:10.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 10 monclient(hunting): picked mon.noname-a con 0x7f8c38142690 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 10 monclient(hunting): picked mon.noname-c con 0x7f8c380b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c38142c80 con 0x7f8c38142690 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c380aae40 con 0x7f8c380b17d0 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3813fbd0 con 0x7f8c381422b0 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3faa5640 10 monclient(hunting): authenticate will time out at 1186.504509s 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3d040640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8c38142690 0x7f8c380b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48286/0 (socket says 172.21.15.64:48286) 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.234+0000 7f8c3d040640 1 -- 172.21.15.64:0/1374217168 learned_addr learned my addr 172.21.15.64:0/1374217168 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014081946 0 0) 0x7f8c38142c80 con 0x7f8c38142690 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c24003520 con 0x7f8c38142690 2024-03-29T11:24:10.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585433480 0 0) 0x7f8c380aae40 con 0x7f8c380b17d0 2024-03-29T11:24:10.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:24:10.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c38142c80 con 0x7f8c380b17d0 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180979772 0 0) 0x7f8c3813fbd0 con 0x7f8c381422b0 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.235+0000 7f8c3c83f640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c240039d0 con 0x7f8c381422b0 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2445155435 0 0) 0x7f8c38142c80 con 0x7f8c380b17d0 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 >> v1:172.21.15.64:6790/0 conn(0x7f8c381422b0 legacy=0x7f8c3813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 >> v1:172.21.15.64:6789/0 conn(0x7f8c38142690 legacy=0x7f8c380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: _finish_hunting 0 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 monclient: found mon.noname-c 2024-03-29T11:24:10.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c38151940 con 0x7f8c380b17d0 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: _finish_auth 0 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: _check_auth_tickets 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c340015f0 con 0x7f8c380b17d0 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3c83f640 10 monclient: dump: 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.221 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3faa5640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.236+0000 7f8c3faa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c34001750 con 0x7f8c380b17d0 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3c83f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3c83f640 1 -- 172.21.15.64:0/1374217168 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c34001b20 con 0x7f8c380b17d0 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3c83f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3c83f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3c83f640 10 monclient: dump: 2024-03-29T11:24:10.222 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3faa5640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.237+0000 7f8c3faa5640 4 set_mon_vals no callback set 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 10 monclient: shutdown 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 1 -- 172.21.15.64:0/1374217168 >> v1:172.21.15.136:6789/0 conn(0x7f8c380b17d0 legacy=0x7f8c380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 1 -- 172.21.15.64:0/1374217168 shutdown_connections 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 1 -- 172.21.15.64:0/1374217168 >> 172.21.15.64:0/1374217168 conn(0x7f8c38066a00 msgr2=0x7f8c38066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 1 -- 172.21.15.64:0/1374217168 shutdown_connections 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.238+0000 7f8c3faa5640 1 -- 172.21.15.64:0/1374217168 wait complete. 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.239+0000 7f8c3faa5640 10 monclient: build_initial_monmap 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.239+0000 7f8c3faa5640 10 monclient: monmap: 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:10.224 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:10.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 1 librados: starting msgr at 2024-03-29T11:24:10.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 1 librados: starting objecter 2024-03-29T11:24:10.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 1 Processor -- start 2024-03-29T11:24:10.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 1 -- start start 2024-03-29T11:24:10.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 1 librados: setting wanted keys 2024-03-29T11:24:10.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 1 librados: calling monclient init 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.240+0000 7f8c3faa5640 10 monclient: init 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.241+0000 7f8c3faa5640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.241+0000 7f8c3faa5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.241+0000 7f8c3faa5640 10 monclient(hunting): picked mon.noname-a con 0x7f8c380b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c3faa5640 10 monclient(hunting): picked mon.noname-b con 0x7f8c381422b0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c3d841640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8c380b17d0 0x7f8c38132580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48444/0 (socket says 172.21.15.64:48444) 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c3d841640 1 -- 172.21.15.64:0/4154534088 learned_addr learned my addr 172.21.15.64:0/4154534088 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c2e7fc640 10 client.?.objecter ms_handle_connect 0x7f8c380b17d0 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c3faa5640 10 monclient(hunting): picked mon.noname-c con 0x7f8c38142690 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c2e7fc640 10 client.?.objecter ms_handle_connect 0x7f8c381422b0 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c3faa5640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c381320f0 con 0x7f8c380b17d0 2024-03-29T11:24:10.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.242+0000 7f8c3faa5640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3812e670 con 0x7f8c381422b0 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c3faa5640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c380ac030 con 0x7f8c38142690 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c3faa5640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c3faa5640 10 monclient(hunting): authenticate will time out at 1186.512958s 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239123546 0 0) 0x7f8c3812e670 con 0x7f8c381422b0 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1c004eb0 con 0x7f8c381422b0 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407231834 0 0) 0x7f8c381320f0 con 0x7f8c380b17d0 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c3812e670 con 0x7f8c380b17d0 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 client.?.objecter ms_handle_connect 0x7f8c38142690 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.243+0000 7f8c2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1426430063 0 0) 0x7f8c1c004eb0 con 0x7f8c381422b0 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c381320f0 con 0x7f8c381422b0 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c28001770 con 0x7f8c381422b0 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.232 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569691118 0 0) 0x7f8c380ac030 con 0x7f8c38142690 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1c004eb0 con 0x7f8c38142690 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2571391426 0 0) 0x7f8c381320f0 con 0x7f8c381422b0 2024-03-29T11:24:10.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 >> v1:172.21.15.64:6790/0 conn(0x7f8c38142690 legacy=0x7f8c3812df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 >> v1:172.21.15.64:6789/0 conn(0x7f8c380b17d0 legacy=0x7f8c38132580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 monclient: found mon.b 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380ac310 con 0x7f8c381422b0 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c3faa5640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c3faa5640 10 monclient: _renew_subs 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c3faa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.244+0000 7f8c3faa5640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c38132c90 con 0x7f8c381422b0 2024-03-29T11:24:10.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.245+0000 7f8c3faa5640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.245+0000 7f8c3faa5640 10 monclient: _renew_subs 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.245+0000 7f8c3faa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.245+0000 7f8c3faa5640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3812ffa0 con 0x7f8c381422b0 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.245+0000 7f8c3faa5640 1 librados: init done 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.246+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c28001ce0 con 0x7f8c381422b0 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.246+0000 7f8c2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.246+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c28004980 con 0x7f8c381422b0 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.246+0000 7f8c23fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.246+0000 7f8c2ffff640 4 set_mon_vals no callback set 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.247+0000 7f8c23fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.247+0000 7f8c23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.247+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c18005080 con 0x7f8c381422b0 2024-03-29T11:24:10.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.247+0000 7f8c2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.247+0000 7f8c2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.248+0000 7f8c2e7fc640 10 monclient: dump: 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:10.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.248+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f8c28004ba0 con 0x7f8c381422b0 2024-03-29T11:24:10.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c3d841640 10 monclient: get_auth_request con 0x7f8c1c06de50 auth_method 0 2024-03-29T11:24:10.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f8c28084920 con 0x7f8c381422b0 2024-03-29T11:24:10.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c2e7fc640 10 client.4240.objecter ms_dispatch 0x7f8c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:10.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c2e7fc640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c2e7fc640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c2e7fc640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:24:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.249+0000 7f8c2e7fc640 10 client.4240.objecter ms_handle_connect 0x7f8c1c06de50 2024-03-29T11:24:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.256+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c28050200 con 0x7f8c381422b0 2024-03-29T11:24:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.256+0000 7f8c2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.256+0000 7f8c2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:10.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.629+0000 7f8c23fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.629+0000 7f8c23fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.629+0000 7f8c23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:24:10.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.629+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c18005330 con 0x7f8c381422b0 2024-03-29T11:24:10.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.631+0000 7f8c2e7fc640 1 -- 172.21.15.64:0/4154534088 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f8c28052130 con 0x7f8c381422b0 2024-03-29T11:24:10.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.631+0000 7f8c2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.631+0000 7f8c2e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:10.612 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:10.612 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.634+0000 7f8c23fff640 10 librados: watch_flush enter 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.634+0000 7f8c23fff640 10 librados: watch_flush exit 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.634+0000 7f8c23fff640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.634+0000 7f8c23fff640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 >> v1:172.21.15.136:6816/2312274743 conn(0x7f8c1c06de50 legacy=0x7f8c1c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 10 monclient: shutdown 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:10.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 >> v1:172.21.15.136:6789/0 conn(0x7f8c381422b0 legacy=0x7f8c3812d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 shutdown_connections 2024-03-29T11:24:10.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 >> 172.21.15.64:0/4154534088 conn(0x7f8c38066a00 msgr2=0x7f8c380b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:10.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 shutdown_connections 2024-03-29T11:24:10.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 -- 172.21.15.64:0/4154534088 wait complete. 2024-03-29T11:24:10.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:10.635+0000 7f8c23fff640 1 librados: shutdown 2024-03-29T11:24:10.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:17.437 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.599+0000 7f9449142640 10 monclient: get_monmap_and_config 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.599+0000 7f9449142640 10 monclient: build_initial_monmap 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.599+0000 7f9449142640 10 monclient: monmap: 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:17.601068+0000 2024-03-29T11:24:17.581 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:17.601068+0000 2024-03-29T11:24:17.582 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:17.582 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.582 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:17.582 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:17.582 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:17.582 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.600+0000 7f9449142640 1 Processor -- start 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.600+0000 7f9449142640 1 -- start start 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.600+0000 7f9449142640 10 monclient: init 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.600+0000 7f9449142640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.600+0000 7f9449142640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.600+0000 7f9449142640 10 monclient(hunting): picked mon.noname-b con 0x7f944406ec80 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 10 monclient(hunting): picked mon.noname-c con 0x7f944406fee0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 10 monclient(hunting): picked mon.noname-a con 0x7f944413f3f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440af000 con 0x7f944413f3f0 2024-03-29T11:24:17.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944406fc20 con 0x7f944406fee0 2024-03-29T11:24:17.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9444071dd0 con 0x7f944406ec80 2024-03-29T11:24:17.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:17.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9449142640 10 monclient(hunting): authenticate will time out at 1193.871949s 2024-03-29T11:24:17.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9443fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f944406ec80 0x7f94440ae8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49982/0 (socket says 172.21.15.64:49982) 2024-03-29T11:24:17.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9443fff640 1 -- 172.21.15.64:0/3801371820 learned_addr learned my addr 172.21.15.64:0/3801371820 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095305089 0 0) 0x7f944406fc20 con 0x7f944406fee0 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f942c003520 con 0x7f944406fee0 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735794083 0 0) 0x7f9444071dd0 con 0x7f944406ec80 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.601+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f942c0048a0 con 0x7f944406ec80 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931060233 0 0) 0x7f9438001110 con 0x7f944413f3f0 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9444071dd0 con 0x7f944413f3f0 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340433925 0 0) 0x7f942c003520 con 0x7f944406fee0 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 >> v1:172.21.15.64:6790/0 conn(0x7f944406ec80 legacy=0x7f94440ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 >> v1:172.21.15.64:6789/0 conn(0x7f944413f3f0 legacy=0x7f944414a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:17.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 monclient: found mon.noname-c 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94440722f0 con 0x7f944406fee0 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f943c001920 con 0x7f944406fee0 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9442ffd640 10 monclient: dump: 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.587 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9449142640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:24:17.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.602+0000 7f9449142640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f943c001a80 con 0x7f944406fee0 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9442ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9442ffd640 1 -- 172.21.15.64:0/3801371820 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f943c001e90 con 0x7f944406fee0 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9442ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9442ffd640 10 monclient: dump: 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.589 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9449142640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.605+0000 7f9449142640 4 set_mon_vals no callback set 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.606+0000 7f9449142640 10 monclient: shutdown 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.606+0000 7f9449142640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.606+0000 7f9449142640 1 -- 172.21.15.64:0/3801371820 >> v1:172.21.15.136:6789/0 conn(0x7f944406fee0 legacy=0x7f944413ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.607+0000 7f9449142640 1 -- 172.21.15.64:0/3801371820 shutdown_connections 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.607+0000 7f9449142640 1 -- 172.21.15.64:0/3801371820 >> 172.21.15.64:0/3801371820 conn(0x7f9444067420 msgr2=0x7f94440b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:17.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.607+0000 7f9449142640 1 -- 172.21.15.64:0/3801371820 shutdown_connections 2024-03-29T11:24:17.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.607+0000 7f9449142640 1 -- 172.21.15.64:0/3801371820 wait complete. 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.608+0000 7f9449142640 10 monclient: build_initial_monmap 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.608+0000 7f9449142640 10 monclient: monmap: 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:17.593 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.608+0000 7f9449142640 1 librados: starting msgr at 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.608+0000 7f9449142640 1 librados: starting objecter 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.608+0000 7f9449142640 1 Processor -- start 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.608+0000 7f9449142640 1 -- start start 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.609+0000 7f9449142640 1 librados: setting wanted keys 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.609+0000 7f9449142640 1 librados: calling monclient init 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.609+0000 7f9449142640 10 monclient: init 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.609+0000 7f9449142640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.609+0000 7f9449142640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9449142640 10 monclient(hunting): picked mon.noname-b con 0x7f944406ec80 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9449142640 10 monclient(hunting): picked mon.noname-c con 0x7f944413f3f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:17.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9443fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f944406ec80 0x7f94441c7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:37306/0 (socket says 172.21.15.64:37306) 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9443fff640 1 -- 172.21.15.64:0/3317824872 learned_addr learned my addr 172.21.15.64:0/3317824872 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9440ff9640 10 client.?.objecter ms_handle_connect 0x7f944406ec80 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9440ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9449142640 10 monclient(hunting): picked mon.noname-a con 0x7f94441c9580 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9440ff9640 10 client.?.objecter ms_handle_connect 0x7f94441c9580 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9440ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9440ff9640 10 client.?.objecter ms_handle_connect 0x7f944413f3f0 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9440ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.610+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94441c6070 con 0x7f94441c9580 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94441c6c60 con 0x7f944406ec80 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94441c6f40 con 0x7f944413f3f0 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9449142640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:17.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9449142640 10 monclient(hunting): authenticate will time out at 1193.882281s 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888186910 0 0) 0x7f94441c6070 con 0x7f94441c9580 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9440ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.611+0000 7f9440ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9430004eb0 con 0x7f94441c9580 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237151826 0 0) 0x7f94441c6c60 con 0x7f944406ec80 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94441c6070 con 0x7f944406ec80 2024-03-29T11:24:17.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549062445 0 0) 0x7f94441c6f40 con 0x7f944413f3f0 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94441c6c60 con 0x7f944413f3f0 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930704542 0 0) 0x7f94441c6c60 con 0x7f944413f3f0 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94441c6f40 con 0x7f944413f3f0 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f943c001d90 con 0x7f944413f3f0 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.612+0000 7f9440ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.601 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226347787 0 0) 0x7f9430004eb0 con 0x7f94441c9580 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94441c6c60 con 0x7f94441c9580 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9438001840 con 0x7f94441c9580 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:17.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2985478911 0 0) 0x7f94441c6f40 con 0x7f944413f3f0 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 >> v1:172.21.15.136:6789/0 conn(0x7f944406ec80 legacy=0x7f94441c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 >> v1:172.21.15.64:6789/0 conn(0x7f94441c9580 legacy=0x7f94441e61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 monclient: found mon.c 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94441c7220 con 0x7f944413f3f0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9440ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9449142640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9449142640 10 monclient: _renew_subs 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9449142640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.613+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94441c7500 con 0x7f944413f3f0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9449142640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9449142640 10 monclient: _renew_subs 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9449142640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94441e7900 con 0x7f944413f3f0 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9449142640 1 librados: init done 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f943c001400 con 0x7f944413f3f0 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9440ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9441ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f943c001ef0 con 0x7f944413f3f0 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9440ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9440ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.614+0000 7f9440ff9640 10 monclient: dump: 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.616+0000 7f9449142640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.616+0000 7f9449142640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.616+0000 7f9449142640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.616+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9424005080 con 0x7f944413f3f0 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f943c019c30 con 0x7f944413f3f0 2024-03-29T11:24:17.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f943c0839b0 con 0x7f944413f3f0 2024-03-29T11:24:17.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9440ff9640 10 client.4244.objecter ms_dispatch 0x7f9444000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:17.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9440ff9640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:17.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9440ff9640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:17.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9443fff640 10 monclient: get_auth_request con 0x7f943006df00 auth_method 0 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.618+0000 7f9440ff9640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.619+0000 7f9440ff9640 10 client.4244.objecter ms_handle_connect 0x7f943006df00 2024-03-29T11:24:17.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.627+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f943c04f2d0 con 0x7f944413f3f0 2024-03-29T11:24:17.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.627+0000 7f9440ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:17.627+0000 7f9440ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:17.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.017+0000 7f9449142640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.017+0000 7f9449142640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.017+0000 7f9449142640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:17.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.017+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9424005330 con 0x7f944413f3f0 2024-03-29T11:24:18.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.018+0000 7f9440ff9640 1 -- 172.21.15.64:0/3317824872 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f943c051200 con 0x7f944413f3f0 2024-03-29T11:24:18.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.018+0000 7f9440ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:18.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.018+0000 7f9440ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:18.000 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:18.000 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.021+0000 7f9449142640 10 librados: watch_flush enter 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 10 librados: watch_flush exit 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 >> v1:172.21.15.136:6816/2312274743 conn(0x7f943006df00 legacy=0x7f94300703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 10 monclient: shutdown 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:18.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 >> v1:172.21.15.64:6790/0 conn(0x7f944413f3f0 legacy=0x7f94441c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 shutdown_connections 2024-03-29T11:24:18.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 >> 172.21.15.64:0/3317824872 conn(0x7f9444067420 msgr2=0x7f9444069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:18.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.022+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 shutdown_connections 2024-03-29T11:24:18.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.023+0000 7f9449142640 1 -- 172.21.15.64:0/3317824872 wait complete. 2024-03-29T11:24:18.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:18.023+0000 7f9449142640 1 librados: shutdown 2024-03-29T11:24:18.019 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:24.826 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.995+0000 7fdb7efc4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.995+0000 7fdb7efc4640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.995+0000 7fdb7efc4640 10 monclient: monmap: 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:24.996100+0000 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:24.996100+0000 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:24.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.996+0000 7fdb7efc4640 1 Processor -- start 2024-03-29T11:24:24.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.996+0000 7fdb7efc4640 1 -- start start 2024-03-29T11:24:24.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.996+0000 7fdb7efc4640 10 monclient: init 2024-03-29T11:24:24.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.996+0000 7fdb7efc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.996+0000 7fdb7efc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:24.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.996+0000 7fdb7efc4640 10 monclient(hunting): picked mon.noname-a con 0x7fdb780b0500 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 10 monclient(hunting): picked mon.noname-b con 0x7fdb780b08e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 10 monclient(hunting): picked mon.noname-c con 0x7fdb780aeca0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb78140620 con 0x7fdb780b0500 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7813daf0 con 0x7fdb780aeca0 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780ae180 con 0x7fdb780b08e0 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:24.999+0000 7fdb7efc4640 10 monclient(hunting): authenticate will time out at 1201.269273s 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.000+0000 7fdb77fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdb780b08e0 0x7fdb7813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48719/0 (socket says 172.21.15.64:48719) 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.000+0000 7fdb77fff640 1 -- 172.21.15.64:0/2246354091 learned_addr learned my addr 172.21.15.64:0/2246354091 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.000+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057972312 0 0) 0x7fdb780ae180 con 0x7fdb780b08e0 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:24:24.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb68003520 con 0x7fdb780b08e0 2024-03-29T11:24:24.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584004985 0 0) 0x7fdb68003520 con 0x7fdb780b08e0 2024-03-29T11:24:24.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 >> v1:172.21.15.136:6789/0 conn(0x7fdb780aeca0 legacy=0x7fdb7814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:24.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 >> v1:172.21.15.64:6789/0 conn(0x7fdb780b0500 legacy=0x7fdb7813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780accf0 con 0x7fdb780b08e0 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdb64001690 con 0x7fdb780b08e0 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: dump: 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:24.984 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb7efc4640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb7efc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb64001860 con 0x7fdb780b08e0 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 1 -- 172.21.15.64:0/2246354091 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdb64001c70 con 0x7fdb780b08e0 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb777fe640 10 monclient: dump: 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb7efc4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:24.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.001+0000 7fdb7efc4640 4 set_mon_vals no callback set 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.002+0000 7fdb7efc4640 10 monclient: shutdown 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.002+0000 7fdb7efc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.002+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2246354091 >> v1:172.21.15.64:6790/0 conn(0x7fdb780b08e0 legacy=0x7fdb7813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.003+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2246354091 shutdown_connections 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.003+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2246354091 >> 172.21.15.64:0/2246354091 conn(0x7fdb7806b660 msgr2=0x7fdb7806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.003+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2246354091 shutdown_connections 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.003+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2246354091 wait complete. 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.004+0000 7fdb7efc4640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.004+0000 7fdb7efc4640 10 monclient: monmap: 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:24.988 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.006+0000 7fdb7efc4640 1 librados: starting msgr at 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.006+0000 7fdb7efc4640 1 librados: starting objecter 2024-03-29T11:24:24.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.006+0000 7fdb7efc4640 1 Processor -- start 2024-03-29T11:24:24.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.007+0000 7fdb7efc4640 1 -- start start 2024-03-29T11:24:24.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.007+0000 7fdb7efc4640 1 librados: setting wanted keys 2024-03-29T11:24:24.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.007+0000 7fdb7efc4640 1 librados: calling monclient init 2024-03-29T11:24:24.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.007+0000 7fdb7efc4640 10 monclient: init 2024-03-29T11:24:24.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.008+0000 7fdb7efc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.008+0000 7fdb7efc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 10 monclient(hunting): picked mon.noname-a con 0x7fdb780aeca0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 10 monclient(hunting): picked mon.noname-b con 0x7fdb7811c9f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 10 monclient(hunting): picked mon.noname-c con 0x7fdb7813af40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7811ec20 con 0x7fdb780aeca0 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7811d5a0 con 0x7fdb7811c9f0 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7806fc40 con 0x7fdb7813af40 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.009+0000 7fdb7efc4640 10 monclient(hunting): authenticate will time out at 1201.279188s 2024-03-29T11:24:24.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.011+0000 7fdb77fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fdb7811c9f0 0x7fdb781d6320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:58016/0 (socket says 172.21.15.64:58016) 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb77fff640 1 -- 172.21.15.64:0/2394425287 learned_addr learned my addr 172.21.15.64:0/2394425287 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb757fa640 10 client.?.objecter ms_handle_connect 0x7fdb780aeca0 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb757fa640 10 client.?.objecter ms_handle_connect 0x7fdb7813af40 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb757fa640 10 client.?.objecter ms_handle_connect 0x7fdb7811c9f0 2024-03-29T11:24:24.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.012+0000 7fdb757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.013+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473613049 0 0) 0x7fdb7811ec20 con 0x7fdb780aeca0 2024-03-29T11:24:24.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.013+0000 7fdb757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.013+0000 7fdb757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.013+0000 7fdb757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.013+0000 7fdb757fa640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:24:24.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.013+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c004eb0 con 0x7fdb780aeca0 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2738261085 0 0) 0x7fdb5c004eb0 con 0x7fdb780aeca0 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb7811ec20 con 0x7fdb780aeca0 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdb6c001930 con 0x7fdb780aeca0 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:24.998 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.014+0000 7fdb757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 482734883 0 0) 0x7fdb7811ec20 con 0x7fdb780aeca0 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 >> v1:172.21.15.64:6790/0 conn(0x7fdb7813af40 legacy=0x7fdb781de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 >> v1:172.21.15.136:6789/0 conn(0x7fdb7811c9f0 legacy=0x7fdb781d6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 1 monclient: found mon.a 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7806fdd0 con 0x7fdb780aeca0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 10 monclient: _renew_subs 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb781d6a30 con 0x7fdb780aeca0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 10 monclient: _renew_subs 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7811ced0 con 0x7fdb780aeca0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.015+0000 7fdb7efc4640 1 librados: init done 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.016+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb6c001a90 con 0x7fdb780aeca0 2024-03-29T11:24:25.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.016+0000 7fdb757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.016+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdb6c0038c0 con 0x7fdb780aeca0 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.016+0000 7fdb757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.016+0000 7fdb757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.016+0000 7fdb757fa640 10 monclient: dump: 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:25.001 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.017+0000 7fdb76ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.018+0000 7fdb7efc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.018+0000 7fdb7efc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.018+0000 7fdb7efc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.018+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb58005080 con 0x7fdb780aeca0 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.019+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fdb6c003a20 con 0x7fdb780aeca0 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb77fff640 10 monclient: get_auth_request con 0x7fdb5c06dde0 auth_method 0 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fdb6c0838d0 con 0x7fdb780aeca0 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb757fa640 10 client.4251.objecter ms_dispatch 0x7fdb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:25.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb757fa640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:25.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb757fa640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:25.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb757fa640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:24:25.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.020+0000 7fdb757fa640 10 client.4251.objecter ms_handle_connect 0x7fdb5c06dde0 2024-03-29T11:24:25.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.029+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb6c04f1b0 con 0x7fdb780aeca0 2024-03-29T11:24:25.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.029+0000 7fdb757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.029+0000 7fdb757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.357+0000 7fdb7efc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.357+0000 7fdb7efc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.357+0000 7fdb7efc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.357+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb58005330 con 0x7fdb780aeca0 2024-03-29T11:24:25.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.358+0000 7fdb757fa640 1 -- 172.21.15.64:0/2394425287 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fdb6c0510e0 con 0x7fdb780aeca0 2024-03-29T11:24:25.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.358+0000 7fdb757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.358+0000 7fdb757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:25.341 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:25.341 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.361+0000 7fdb7efc4640 10 librados: watch_flush enter 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.361+0000 7fdb7efc4640 10 librados: watch_flush exit 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.361+0000 7fdb7efc4640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.361+0000 7fdb7efc4640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.362+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 >> v1:172.21.15.136:6816/2312274743 conn(0x7fdb5c06dde0 legacy=0x7fdb5c0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.362+0000 7fdb7efc4640 10 monclient: shutdown 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.362+0000 7fdb7efc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:25.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.362+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 >> v1:172.21.15.64:6789/0 conn(0x7fdb780aeca0 legacy=0x7fdb7813a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.362+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 shutdown_connections 2024-03-29T11:24:25.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.362+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 >> 172.21.15.64:0/2394425287 conn(0x7fdb7806b660 msgr2=0x7fdb780a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:25.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.363+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 shutdown_connections 2024-03-29T11:24:25.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.364+0000 7fdb7efc4640 1 -- 172.21.15.64:0/2394425287 wait complete. 2024-03-29T11:24:25.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:25.364+0000 7fdb7efc4640 1 librados: shutdown 2024-03-29T11:24:25.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:32.168 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.336+0000 7f5d82102640 10 monclient: get_monmap_and_config 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.336+0000 7f5d82102640 10 monclient: build_initial_monmap 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.336+0000 7f5d82102640 10 monclient: monmap: 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:32.337285+0000 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:32.337285+0000 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.337+0000 7f5d82102640 1 Processor -- start 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.337+0000 7f5d82102640 1 -- start start 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.337+0000 7f5d82102640 10 monclient: init 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7c06c250 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7c072400 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7c06c630 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06d760 con 0x7f5d7c06c630 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c070500 con 0x7f5d7c06c250 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0aff50 con 0x7f5d7c072400 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.338+0000 7f5d82102640 10 monclient(hunting): authenticate will time out at 1208.608463s 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.341+0000 7f5d7b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5d7c06c250 0x7f5d7c071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:54316/0 (socket says 172.21.15.64:54316) 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.341+0000 7f5d7b7fe640 1 -- 172.21.15.64:0/3913101525 learned_addr learned my addr 172.21.15.64:0/3913101525 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552901132 0 0) 0x7f5d7c070500 con 0x7f5d7c06c250 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d5c003520 con 0x7f5d7c06c250 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085475745 0 0) 0x7f5d5c003520 con 0x7f5d7c06c250 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 >> v1:172.21.15.64:6790/0 conn(0x7f5d7c072400 legacy=0x7f5d7c1465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.342+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 >> v1:172.21.15.64:6789/0 conn(0x7f5d7c06c630 legacy=0x7f5d7c151ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 1 monclient: found mon.noname-c 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c0b00e0 con 0x7f5d7c06c250 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d82102640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d82102640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5d6c001690 con 0x7f5d7c06c250 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d7affd640 10 monclient: dump: 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.345+0000 7f5d82102640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.346+0000 7f5d7affd640 1 -- 172.21.15.64:0/3913101525 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d6c001920 con 0x7f5d7c06c250 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.346+0000 7f5d7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.346+0000 7f5d82102640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.346+0000 7f5d82102640 4 set_mon_vals no callback set 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.347+0000 7f5d82102640 10 monclient: shutdown 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.347+0000 7f5d82102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.347+0000 7f5d82102640 1 -- 172.21.15.64:0/3913101525 >> v1:172.21.15.136:6789/0 conn(0x7f5d7c06c250 legacy=0x7f5d7c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.349+0000 7f5d82102640 1 -- 172.21.15.64:0/3913101525 shutdown_connections 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.349+0000 7f5d82102640 1 -- 172.21.15.64:0/3913101525 >> 172.21.15.64:0/3913101525 conn(0x7f5d7c067610 msgr2=0x7f5d7c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.350+0000 7f5d82102640 1 -- 172.21.15.64:0/3913101525 shutdown_connections 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.350+0000 7f5d82102640 1 -- 172.21.15.64:0/3913101525 wait complete. 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.351+0000 7f5d82102640 10 monclient: build_initial_monmap 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.351+0000 7f5d82102640 10 monclient: monmap: 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.351+0000 7f5d82102640 1 librados: starting msgr at 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.351+0000 7f5d82102640 1 librados: starting objecter 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 1 Processor -- start 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 1 -- start start 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 1 librados: setting wanted keys 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 1 librados: calling monclient init 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 10 monclient: init 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.352+0000 7f5d82102640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7c06c250 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7c06c630 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7c072400 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c1db4b0 con 0x7f5d7c06c250 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0abed0 con 0x7f5d7c072400 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0a9ce0 con 0x7f5d7c06c630 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d82102640 10 monclient(hunting): authenticate will time out at 1208.625356s 2024-03-29T11:24:32.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d73fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5d7c06c630 0x7f5d7c0aa210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36193/0 (socket says 172.21.15.64:36193) 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d73fff640 1 -- 172.21.15.64:0/671485715 learned_addr learned my addr 172.21.15.64:0/671485715 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d78ff9640 10 client.?.objecter ms_handle_connect 0x7f5d7c06c630 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.355+0000 7f5d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 client.?.objecter ms_handle_connect 0x7f5d7c072400 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198509036 0 0) 0x7f5d7c0a9ce0 con 0x7f5d7c06c630 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60004eb0 con 0x7f5d7c06c630 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512138023 0 0) 0x7f5d7c0abed0 con 0x7f5d7c072400 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.356+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c0a9ce0 con 0x7f5d7c072400 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078058610 0 0) 0x7f5d60004eb0 con 0x7f5d7c06c630 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d7c0abed0 con 0x7f5d7c06c630 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5d64001990 con 0x7f5d7c06c630 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.345 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3966675882 0 0) 0x7f5d7c0abed0 con 0x7f5d7c06c630 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 >> v1:172.21.15.136:6789/0 conn(0x7f5d7c072400 legacy=0x7f5d7c1e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.357+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 >> v1:172.21.15.64:6789/0 conn(0x7f5d7c06c250 legacy=0x7f5d7c1db940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.358+0000 7f5d78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.358+0000 7f5d78ff9640 1 monclient: found mon.c 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.358+0000 7f5d78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.358+0000 7f5d78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.358+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c0b3610 con 0x7f5d7c06c630 2024-03-29T11:24:32.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 10 monclient: _renew_subs 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c1dc050 con 0x7f5d7c06c630 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 10 monclient: _renew_subs 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c0aab00 con 0x7f5d7c06c630 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.359+0000 7f5d82102640 1 librados: init done 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d64001af0 con 0x7f5d7c06c630 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:32.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5d64003a60 con 0x7f5d7c06c630 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.360+0000 7f5d78ff9640 10 monclient: dump: 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.361+0000 7f5d82102640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.361+0000 7f5d82102640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.361+0000 7f5d82102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.361+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d54005080 con 0x7f5d7c06c630 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.363+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f5d64003bc0 con 0x7f5d7c06c630 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.364+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f5d64084a90 con 0x7f5d7c06c630 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.364+0000 7f5d78ff9640 10 client.4253.objecter ms_dispatch 0x7f5d7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.364+0000 7f5d78ff9640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:32.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.364+0000 7f5d78ff9640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:32.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.364+0000 7f5d78ff9640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:24:32.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.364+0000 7f5d7b7fe640 10 monclient: get_auth_request con 0x7f5d6006dd00 auth_method 0 2024-03-29T11:24:32.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.365+0000 7f5d78ff9640 10 client.4253.objecter ms_handle_connect 0x7f5d6006dd00 2024-03-29T11:24:32.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.371+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d640501a0 con 0x7f5d7c06c630 2024-03-29T11:24:32.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.371+0000 7f5d78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.371+0000 7f5d78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:32.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.714+0000 7f5d82102640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.714+0000 7f5d82102640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.714+0000 7f5d82102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:32.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.714+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d54005330 con 0x7f5d7c06c630 2024-03-29T11:24:32.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.716+0000 7f5d78ff9640 1 -- 172.21.15.64:0/671485715 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f5d640522a0 con 0x7f5d7c06c630 2024-03-29T11:24:32.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.716+0000 7f5d78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.716+0000 7f5d78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:32.699 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:32.699 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:32.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 10 librados: watch_flush enter 2024-03-29T11:24:32.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 10 librados: watch_flush exit 2024-03-29T11:24:32.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:24:32.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 >> v1:172.21.15.136:6816/2312274743 conn(0x7f5d6006dd00 legacy=0x7f5d600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 10 monclient: shutdown 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 >> v1:172.21.15.64:6790/0 conn(0x7f5d7c06c630 legacy=0x7f5d7c0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 shutdown_connections 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.720+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 >> 172.21.15.64:0/671485715 conn(0x7f5d7c067610 msgr2=0x7f5d7c143740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.721+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 shutdown_connections 2024-03-29T11:24:32.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.721+0000 7f5d82102640 1 -- 172.21.15.64:0/671485715 wait complete. 2024-03-29T11:24:32.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:32.722+0000 7f5d82102640 1 librados: shutdown 2024-03-29T11:24:32.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:39.529 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:39.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.700+0000 7fd76bcc8640 10 monclient: get_monmap_and_config 2024-03-29T11:24:39.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.700+0000 7fd76bcc8640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.700+0000 7fd76bcc8640 10 monclient: monmap: 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:39.701403+0000 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:39.701403+0000 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:39.683 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:39.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.701+0000 7fd76bcc8640 1 Processor -- start 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 1 -- start start 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient: init 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient(hunting): picked mon.noname-a con 0x7fd7640b1f60 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient(hunting): picked mon.noname-b con 0x7fd764071eb0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient(hunting): picked mon.noname-c con 0x7fd764143d10 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76406ef80 con 0x7fd7640b1f60 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76406f260 con 0x7fd764143d10 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640b40c0 con 0x7fd764071eb0 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.702+0000 7fd76bcc8640 10 monclient(hunting): authenticate will time out at 1215.972769s 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.706+0000 7fd769263640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd764071eb0 0x7fd764146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56695/0 (socket says 172.21.15.64:56695) 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.706+0000 7fd769263640 1 -- 172.21.15.64:0/1151384668 learned_addr learned my addr 172.21.15.64:0/1151384668 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529605019 0 0) 0x7fd7640b40c0 con 0x7fd764071eb0 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd750003520 con 0x7fd764071eb0 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639333180 0 0) 0x7fd76406f260 con 0x7fd764143d10 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.709+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7640b40c0 con 0x7fd764143d10 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125014546 0 0) 0x7fd76406ef80 con 0x7fd7640b1f60 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd76406f260 con 0x7fd7640b1f60 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098372508 0 0) 0x7fd7640b40c0 con 0x7fd764143d10 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 >> v1:172.21.15.64:6790/0 conn(0x7fd764071eb0 legacy=0x7fd764146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 >> v1:172.21.15.64:6789/0 conn(0x7fd7640b1f60 legacy=0x7fd764143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 monclient: found mon.noname-c 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764155d70 con 0x7fd764143d10 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd76bcc8640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd76bcc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd7600015a0 con 0x7fd764143d10 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: dump: 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.695 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd76bcc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7600017c0 con 0x7fd764143d10 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd76bcc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd76bcc8640 4 set_mon_vals no callback set 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 1 -- 172.21.15.64:0/1151384668 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd760001b50 con 0x7fd764143d10 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.710+0000 7fd768a62640 10 monclient: dump: 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:39.696 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.711+0000 7fd76bcc8640 10 monclient: shutdown 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.711+0000 7fd76bcc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.711+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1151384668 >> v1:172.21.15.136:6789/0 conn(0x7fd764143d10 legacy=0x7fd764146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.711+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1151384668 shutdown_connections 2024-03-29T11:24:39.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.711+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1151384668 >> 172.21.15.64:0/1151384668 conn(0x7fd764066a00 msgr2=0x7fd764066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.712+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1151384668 shutdown_connections 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.713+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1151384668 wait complete. 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.713+0000 7fd76bcc8640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.713+0000 7fd76bcc8640 10 monclient: monmap: 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:39.699 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 1 librados: starting msgr at 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 1 librados: starting objecter 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 1 Processor -- start 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 1 -- start start 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 1 librados: setting wanted keys 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 1 librados: calling monclient init 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.714+0000 7fd76bcc8640 10 monclient: init 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient(hunting): picked mon.noname-c con 0x7fd7640b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient(hunting): picked mon.noname-a con 0x7fd764143d10 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient(hunting): picked mon.noname-b con 0x7fd7640abb90 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640ab500 con 0x7fd764143d10 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd764140720 con 0x7fd7640abb90 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640aceb0 con 0x7fd7640b1f60 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.715+0000 7fd76bcc8640 10 monclient(hunting): authenticate will time out at 1215.985467s 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.718+0000 7fd769a64640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd7640b1f60 0x7fd7641e3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56779/0 (socket says 172.21.15.64:56779) 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.718+0000 7fd769a64640 1 -- 172.21.15.64:0/1510868922 learned_addr learned my addr 172.21.15.64:0/1510868922 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.718+0000 7fd75a7fc640 10 client.?.objecter ms_handle_connect 0x7fd7640b1f60 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.718+0000 7fd75a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647145072 0 0) 0x7fd7640aceb0 con 0x7fd7640b1f60 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd748004eb0 con 0x7fd7640b1f60 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115801775 0 0) 0x7fd748004eb0 con 0x7fd7640b1f60 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7640aceb0 con 0x7fd7640b1f60 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd75c001e20 con 0x7fd7640b1f60 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.719+0000 7fd75a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.720+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2410576266 0 0) 0x7fd7640aceb0 con 0x7fd7640b1f60 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.720+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 >> v1:172.21.15.136:6789/0 conn(0x7fd7640abb90 legacy=0x7fd764140010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 >> v1:172.21.15.64:6789/0 conn(0x7fd764143d10 legacy=0x7fd76413f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 1 monclient: found mon.c 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640aaf30 con 0x7fd7640b1f60 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 client.?.objecter ms_handle_connect 0x7fd764143d10 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 client.?.objecter ms_handle_connect 0x7fd7640abb90 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd75a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 10 monclient: _renew_subs 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7640708e0 con 0x7fd7640b1f60 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 10 monclient: _renew_subs 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7641e4420 con 0x7fd7640b1f60 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.723+0000 7fd76bcc8640 1 librados: init done 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd75c001f80 con 0x7fd7640b1f60 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd76bcc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd76bcc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd76bcc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd75c003ea0 con 0x7fd7640b1f60 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7640ab2f0 con 0x7fd7640b1f60 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75a7fc640 10 monclient: dump: 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.726+0000 7fd75bfff640 4 set_mon_vals no callback set 2024-03-29T11:24:39.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.728+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fd75c004080 con 0x7fd7640b1f60 2024-03-29T11:24:39.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.729+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fd75c054db0 con 0x7fd7640b1f60 2024-03-29T11:24:39.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.729+0000 7fd75a7fc640 10 client.4259.objecter ms_dispatch 0x7fd764000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:39.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.729+0000 7fd75a7fc640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:39.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.729+0000 7fd75a7fc640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:39.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.729+0000 7fd75a7fc640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:24:39.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.730+0000 7fd769263640 10 monclient: get_auth_request con 0x7fd74806dc60 auth_method 0 2024-03-29T11:24:39.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.730+0000 7fd75a7fc640 10 client.4259.objecter ms_handle_connect 0x7fd74806dc60 2024-03-29T11:24:39.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.736+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd75c051050 con 0x7fd7640b1f60 2024-03-29T11:24:39.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.736+0000 7fd75a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:39.736+0000 7fd75a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.073+0000 7fd76bcc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.073+0000 7fd76bcc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.073+0000 7fd76bcc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:24:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.073+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7640accb0 con 0x7fd7640b1f60 2024-03-29T11:24:40.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.081+0000 7fd75a7fc640 1 -- 172.21.15.64:0/1510868922 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fd75c0511f0 con 0x7fd7640b1f60 2024-03-29T11:24:40.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.081+0000 7fd75a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.081+0000 7fd75a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:40.064 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:40.064 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:40.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.084+0000 7fd76bcc8640 10 librados: watch_flush enter 2024-03-29T11:24:40.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.084+0000 7fd76bcc8640 10 librados: watch_flush exit 2024-03-29T11:24:40.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.084+0000 7fd76bcc8640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:24:40.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.084+0000 7fd76bcc8640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 >> v1:172.21.15.136:6816/2312274743 conn(0x7fd74806dc60 legacy=0x7fd748070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 10 monclient: shutdown 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 >> v1:172.21.15.64:6790/0 conn(0x7fd7640b1f60 legacy=0x7fd7641e3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 shutdown_connections 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 >> 172.21.15.64:0/1510868922 conn(0x7fd764066a00 msgr2=0x7fd7640b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 shutdown_connections 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 -- 172.21.15.64:0/1510868922 wait complete. 2024-03-29T11:24:40.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:40.085+0000 7fd76bcc8640 1 librados: shutdown 2024-03-29T11:24:40.084 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:46.891 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.070+0000 7f026dee4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.070+0000 7f026dee4640 10 monclient: build_initial_monmap 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.070+0000 7f026dee4640 10 monclient: monmap: 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:47.071446+0000 2024-03-29T11:24:47.055 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:47.071446+0000 2024-03-29T11:24:47.056 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:47.056 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.056 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:47.056 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:47.056 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:47.056 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.074+0000 7f026dee4640 1 Processor -- start 2024-03-29T11:24:47.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 1 -- start start 2024-03-29T11:24:47.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient: init 2024-03-29T11:24:47.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:47.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient(hunting): picked mon.noname-b con 0x7f02681457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient(hunting): picked mon.noname-a con 0x7f026813f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient(hunting): picked mon.noname-c con 0x7f02680a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680aaef0 con 0x7f026813f650 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0268145bd0 con 0x7f02680a9d70 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0268145eb0 con 0x7f02681457f0 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:47.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.076+0000 7f026dee4640 10 monclient(hunting): authenticate will time out at 1223.346837s 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.083+0000 7f0267fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f02680a9d70 0x7f026814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:41501/0 (socket says 172.21.15.64:41501) 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.083+0000 7f0267fff640 1 -- 172.21.15.64:0/1747435899 learned_addr learned my addr 172.21.15.64:0/1747435899 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761532664 0 0) 0x7f0268145bd0 con 0x7f02680a9d70 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250003520 con 0x7f02680a9d70 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705391652 0 0) 0x7f0250003520 con 0x7f02680a9d70 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 >> v1:172.21.15.64:6790/0 conn(0x7f02681457f0 legacy=0x7f026813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 >> v1:172.21.15.64:6789/0 conn(0x7f026813f650 legacy=0x7f02680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.085+0000 7f02667fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268146340 con 0x7f02680a9d70 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f025c001690 con 0x7f02680a9d70 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f02667fc640 10 monclient: dump: 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f026dee4640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.086+0000 7f026dee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.087+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f025c001860 con 0x7f02680a9d70 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.087+0000 7f02667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.087+0000 7f02667fc640 1 -- 172.21.15.64:0/1747435899 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f025c001cb0 con 0x7f02680a9d70 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.087+0000 7f02667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.087+0000 7f02667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.087+0000 7f02667fc640 10 monclient: dump: 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.088+0000 7f026dee4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.088+0000 7f026dee4640 4 set_mon_vals no callback set 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.089+0000 7f026dee4640 10 monclient: shutdown 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.089+0000 7f026dee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.089+0000 7f026dee4640 1 -- 172.21.15.64:0/1747435899 >> v1:172.21.15.136:6789/0 conn(0x7f02680a9d70 legacy=0x7f026814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.090+0000 7f026dee4640 1 -- 172.21.15.64:0/1747435899 shutdown_connections 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.090+0000 7f026dee4640 1 -- 172.21.15.64:0/1747435899 >> 172.21.15.64:0/1747435899 conn(0x7f0268066a00 msgr2=0x7f0268066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.091+0000 7f026dee4640 1 -- 172.21.15.64:0/1747435899 shutdown_connections 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.091+0000 7f026dee4640 1 -- 172.21.15.64:0/1747435899 wait complete. 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.092+0000 7f026dee4640 10 monclient: build_initial_monmap 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.092+0000 7f026dee4640 10 monclient: monmap: 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.092+0000 7f026dee4640 1 librados: starting msgr at 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.092+0000 7f026dee4640 1 librados: starting objecter 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.092+0000 7f026dee4640 1 Processor -- start 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.094+0000 7f026dee4640 1 -- start start 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.094+0000 7f026dee4640 1 librados: setting wanted keys 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.094+0000 7f026dee4640 1 librados: calling monclient init 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.094+0000 7f026dee4640 10 monclient: init 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient(hunting): picked mon.noname-b con 0x7f02680a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient(hunting): picked mon.noname-c con 0x7f026813f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient(hunting): picked mon.noname-a con 0x7f02681457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680ab8e0 con 0x7f02681457f0 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02681dba60 con 0x7f02680a9d70 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0268071730 con 0x7f026813f650 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.095+0000 7f026dee4640 10 monclient(hunting): authenticate will time out at 1223.365082s 2024-03-29T11:24:47.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.100+0000 7f0266ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f026813f650 0x7f0268071020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37569/0 (socket says 172.21.15.64:37569) 2024-03-29T11:24:47.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.100+0000 7f0266ffd640 1 -- 172.21.15.64:0/455796052 learned_addr learned my addr 172.21.15.64:0/455796052 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:47.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.100+0000 7f024ffff640 10 client.?.objecter ms_handle_connect 0x7f026813f650 2024-03-29T11:24:47.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.100+0000 7f024ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.100+0000 7f024ffff640 10 client.?.objecter ms_handle_connect 0x7f02681457f0 2024-03-29T11:24:47.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.100+0000 7f024ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.104+0000 7f024ffff640 10 client.?.objecter ms_handle_connect 0x7f02680a9d70 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.104+0000 7f024ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315709818 0 0) 0x7f02680ab8e0 con 0x7f02681457f0 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0254004eb0 con 0x7f02681457f0 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204566934 0 0) 0x7f0268071730 con 0x7f026813f650 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:24:47.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.105+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02680ab8e0 con 0x7f026813f650 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3810832799 0 0) 0x7f0254004eb0 con 0x7f02681457f0 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0268071730 con 0x7f02681457f0 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f025c0019d0 con 0x7f02681457f0 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.090 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.091 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:47.091 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:47.091 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:47.091 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.106+0000 7f024ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.107+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694948979 0 0) 0x7f02681dba60 con 0x7f02680a9d70 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.107+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.107+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.107+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.107+0000 7f024ffff640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.107+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0254004eb0 con 0x7f02680a9d70 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157628742 0 0) 0x7f02680ab8e0 con 0x7f026813f650 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02681dba60 con 0x7f026813f650 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963209128 0 0) 0x7f0254004eb0 con 0x7f02680a9d70 2024-03-29T11:24:47.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02680ab8e0 con 0x7f02680a9d70 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0260001cb0 con 0x7f026813f650 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:47.095 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f02580016b0 con 0x7f02680a9d70 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.110+0000 7f024ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607365341 0 0) 0x7f0268071730 con 0x7f02681457f0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 >> v1:172.21.15.64:6790/0 conn(0x7f026813f650 legacy=0x7f0268071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 >> v1:172.21.15.136:6789/0 conn(0x7f02680a9d70 legacy=0x7f02681dbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 1 monclient: found mon.a 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02681db730 con 0x7f02681457f0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f024ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 10 monclient: _renew_subs 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026813af50 con 0x7f02681457f0 2024-03-29T11:24:47.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:47.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 10 monclient: _renew_subs 2024-03-29T11:24:47.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:47.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0268070350 con 0x7f02681457f0 2024-03-29T11:24:47.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.111+0000 7f026dee4640 1 librados: init done 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.112+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f025c001c70 con 0x7f02681457f0 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.112+0000 7f024ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.112+0000 7f0265ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.113+0000 7f026dee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.113+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f025c004d90 con 0x7f02681457f0 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.113+0000 7f024ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.113+0000 7f024ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.113+0000 7f024ffff640 10 monclient: dump: 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.099 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:47.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.116+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f025c01ad60 con 0x7f02681457f0 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f026dee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f026dee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0248005080 con 0x7f02681457f0 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f025c084a70 con 0x7f02681457f0 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f024ffff640 10 client.4260.objecter ms_dispatch 0x7f0268000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f024ffff640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f024ffff640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f024ffff640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:24:47.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.117+0000 7f02677fe640 10 monclient: get_auth_request con 0x7f025406e220 auth_method 0 2024-03-29T11:24:47.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.118+0000 7f024ffff640 10 client.4260.objecter ms_handle_connect 0x7f025406e220 2024-03-29T11:24:47.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.125+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f025c050350 con 0x7f02681457f0 2024-03-29T11:24:47.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.125+0000 7f024ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.125+0000 7f024ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:47.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.454+0000 7f026dee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.454+0000 7f026dee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.454+0000 7f026dee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:47.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.454+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0248005330 con 0x7f02681457f0 2024-03-29T11:24:47.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.455+0000 7f024ffff640 1 -- 172.21.15.64:0/455796052 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f025c052280 con 0x7f02681457f0 2024-03-29T11:24:47.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.455+0000 7f024ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.455+0000 7f024ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:47.444 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:47.444 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.462+0000 7f026dee4640 10 librados: watch_flush enter 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.462+0000 7f026dee4640 10 librados: watch_flush exit 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.462+0000 7f026dee4640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.462+0000 7f026dee4640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.463+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 >> v1:172.21.15.136:6816/2312274743 conn(0x7f025406e220 legacy=0x7f02540706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.463+0000 7f026dee4640 10 monclient: shutdown 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.463+0000 7f026dee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:47.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.463+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 >> v1:172.21.15.64:6789/0 conn(0x7f02681457f0 legacy=0x7f026813a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.463+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 shutdown_connections 2024-03-29T11:24:47.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.463+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 >> 172.21.15.64:0/455796052 conn(0x7f0268066a00 msgr2=0x7f02680b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:47.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.465+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 shutdown_connections 2024-03-29T11:24:47.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.466+0000 7f026dee4640 1 -- 172.21.15.64:0/455796052 wait complete. 2024-03-29T11:24:47.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:47.466+0000 7f026dee4640 1 librados: shutdown 2024-03-29T11:24:47.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:54.279 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:54.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.452+0000 7f5a24c63640 10 monclient: get_monmap_and_config 2024-03-29T11:24:54.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.452+0000 7f5a24c63640 10 monclient: build_initial_monmap 2024-03-29T11:24:54.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.452+0000 7f5a24c63640 10 monclient: monmap: 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:24:54.453558+0000 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:24:54.453558+0000 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:24:54.436 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:54.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.454+0000 7f5a24c63640 1 Processor -- start 2024-03-29T11:24:54.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.454+0000 7f5a24c63640 1 -- start start 2024-03-29T11:24:54.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.454+0000 7f5a24c63640 10 monclient: init 2024-03-29T11:24:54.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.455+0000 7f5a24c63640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:54.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.455+0000 7f5a24c63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:54.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.455+0000 7f5a24c63640 10 monclient(hunting): picked mon.noname-b con 0x7f5a200a9750 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:54.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.457+0000 7f5a24c63640 10 monclient(hunting): picked mon.noname-c con 0x7f5a200add40 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:54.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.458+0000 7f5a1f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5a200a9750 0x7f5a20149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:60039/0 (socket says 172.21.15.64:60039) 2024-03-29T11:24:54.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a24c63640 10 monclient(hunting): picked mon.noname-a con 0x7f5a200aad20 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:54.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a1f7fe640 1 -- 172.21.15.64:0/1044297072 learned_addr learned my addr 172.21.15.64:0/1044297072 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:54.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a20149870 con 0x7f5a200aad20 2024-03-29T11:24:54.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a200abf80 con 0x7f5a200add40 2024-03-29T11:24:54.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a20149ac0 con 0x7f5a200a9750 2024-03-29T11:24:54.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a24c63640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:54.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.459+0000 7f5a24c63640 10 monclient(hunting): authenticate will time out at 1230.729825s 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766329236 0 0) 0x7f5a20149ac0 con 0x7f5a200a9750 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a0c003520 con 0x7f5a200a9750 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075026218 0 0) 0x7f5a200abf80 con 0x7f5a200add40 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:24:54.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a20149ac0 con 0x7f5a200add40 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428342180 0 0) 0x7f5a20149ac0 con 0x7f5a200add40 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 >> v1:172.21.15.64:6790/0 conn(0x7f5a200a9750 legacy=0x7f5a20149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 >> v1:172.21.15.64:6789/0 conn(0x7f5a200aad20 legacy=0x7f5a20151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 1 monclient: found mon.noname-c 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.461+0000 7f5a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:24:54.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2014a6f0 con 0x7f5a200add40 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5a08001670 con 0x7f5a200add40 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: dump: 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:54.446 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a08001890 con 0x7f5a200add40 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 1 -- 172.21.15.64:0/1044297072 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5a08001c20 con 0x7f5a200add40 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.462+0000 7f5a1effd640 10 monclient: dump: 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.447 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:54.448 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:54.448 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:54.448 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:54.448 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:54.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.465+0000 7f5a24c63640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:24:54.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.465+0000 7f5a24c63640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:54.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.465+0000 7f5a24c63640 4 set_mon_vals no callback set 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 10 monclient: shutdown 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 >> v1:172.21.15.136:6789/0 conn(0x7f5a200add40 legacy=0x7f5a20143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 shutdown_connections 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 >> 172.21.15.64:0/1044297072 conn(0x7f5a200673e0 msgr2=0x7f5a200b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 shutdown_connections 2024-03-29T11:24:54.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.466+0000 7f5a24c63640 1 -- 172.21.15.64:0/1044297072 wait complete. 2024-03-29T11:24:54.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.468+0000 7f5a24c63640 10 monclient: build_initial_monmap 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.468+0000 7f5a24c63640 10 monclient: monmap: 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.468+0000 7f5a24c63640 1 librados: starting msgr at 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.468+0000 7f5a24c63640 1 librados: starting objecter 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.469+0000 7f5a24c63640 1 Processor -- start 2024-03-29T11:24:54.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.469+0000 7f5a24c63640 1 -- start start 2024-03-29T11:24:54.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.469+0000 7f5a24c63640 1 librados: setting wanted keys 2024-03-29T11:24:54.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.469+0000 7f5a24c63640 1 librados: calling monclient init 2024-03-29T11:24:54.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.469+0000 7f5a24c63640 10 monclient: init 2024-03-29T11:24:54.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.470+0000 7f5a24c63640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:54.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.470+0000 7f5a24c63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:54.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.470+0000 7f5a24c63640 10 monclient(hunting): picked mon.noname-b con 0x7f5a200a9750 addr v1:172.21.15.136:6789/0 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 10 monclient(hunting): picked mon.noname-a con 0x7f5a200aad20 addr v1:172.21.15.64:6789/0 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 10 monclient(hunting): picked mon.noname-c con 0x7f5a200add40 addr v1:172.21.15.64:6790/0 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2006e8c0 con 0x7f5a200aad20 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2006f210 con 0x7f5a200a9750 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a20070330 con 0x7f5a200add40 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.471+0000 7f5a24c63640 10 monclient(hunting): authenticate will time out at 1230.741715s 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.475+0000 7f5a1f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5a200a9750 0x7f5a2006e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:58713/0 (socket says 172.21.15.64:58713) 2024-03-29T11:24:54.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.475+0000 7f5a1f7fe640 1 -- 172.21.15.64:0/95947787 learned_addr learned my addr 172.21.15.64:0/95947787 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 10 client.?.objecter ms_handle_connect 0x7f5a200aad20 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502846591 0 0) 0x7f5a2006e8c0 con 0x7f5a200aad20 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00004eb0 con 0x7f5a200aad20 2024-03-29T11:24:54.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3911979097 0 0) 0x7f5a00004eb0 con 0x7f5a200aad20 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a2006e8c0 con 0x7f5a200aad20 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.476+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5a08001be0 con 0x7f5a200aad20 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:54.465 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 client.?.objecter ms_handle_connect 0x7f5a200add40 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661269993 0 0) 0x7f5a2006e8c0 con 0x7f5a200aad20 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 >> v1:172.21.15.64:6790/0 conn(0x7f5a200add40 legacy=0x7f5a20070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 >> v1:172.21.15.136:6789/0 conn(0x7f5a200a9750 legacy=0x7f5a2006e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 monclient: found mon.a 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2006eb80 con 0x7f5a200aad20 2024-03-29T11:24:54.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 10 monclient: _renew_subs 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2006edd0 con 0x7f5a200aad20 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 10 monclient: _renew_subs 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2013eb60 con 0x7f5a200aad20 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a24c63640 1 librados: init done 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a08001d40 con 0x7f5a200aad20 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:54.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5a08003b00 con 0x7f5a200aad20 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1cff9640 10 monclient: dump: 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:24:54.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.477+0000 7f5a1e7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.479+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f5a08003c60 con 0x7f5a200aad20 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f5a08083980 con 0x7f5a200aad20 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a1cff9640 10 client.4263.objecter ms_dispatch 0x7f5a20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a1cff9640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a1cff9640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a1cff9640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a1f7fe640 10 monclient: get_auth_request con 0x7f5a0006dbd0 auth_method 0 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a24c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.480+0000 7f5a24c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.481+0000 7f5a24c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.481+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a04005080 con 0x7f5a200aad20 2024-03-29T11:24:54.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.481+0000 7f5a1cff9640 10 client.4263.objecter ms_handle_connect 0x7f5a0006dbd0 2024-03-29T11:24:54.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.494+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a0804f090 con 0x7f5a200aad20 2024-03-29T11:24:54.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.494+0000 7f5a1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.494+0000 7f5a1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:54.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.813+0000 7f5a24c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.813+0000 7f5a24c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.813+0000 7f5a24c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:24:54.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.813+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a04005330 con 0x7f5a200aad20 2024-03-29T11:24:54.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.814+0000 7f5a1cff9640 1 -- 172.21.15.64:0/95947787 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f5a08051190 con 0x7f5a200aad20 2024-03-29T11:24:54.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.814+0000 7f5a1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.814+0000 7f5a1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:54.798 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:24:54.799 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:54.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.818+0000 7f5a24c63640 10 librados: watch_flush enter 2024-03-29T11:24:54.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.818+0000 7f5a24c63640 10 librados: watch_flush exit 2024-03-29T11:24:54.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.818+0000 7f5a24c63640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:24:54.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.818+0000 7f5a24c63640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:24:54.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.820+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 >> v1:172.21.15.136:6816/2312274743 conn(0x7f5a0006dbd0 legacy=0x7f5a00070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.820+0000 7f5a24c63640 10 monclient: shutdown 2024-03-29T11:24:54.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.820+0000 7f5a24c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.820+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 >> v1:172.21.15.64:6789/0 conn(0x7f5a200aad20 legacy=0x7f5a2013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.821+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 shutdown_connections 2024-03-29T11:24:54.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.821+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 >> 172.21.15.64:0/95947787 conn(0x7f5a200673e0 msgr2=0x7f5a200aa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.821+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 shutdown_connections 2024-03-29T11:24:54.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.821+0000 7f5a24c63640 1 -- 172.21.15.64:0/95947787 wait complete. 2024-03-29T11:24:54.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:24:54.821+0000 7f5a24c63640 1 librados: shutdown 2024-03-29T11:24:54.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:01.627 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.793+0000 7f48639f6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.793+0000 7f48639f6640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.793+0000 7f48639f6640 10 monclient: monmap: 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:01.794844+0000 2024-03-29T11:25:01.777 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:01.794844+0000 2024-03-29T11:25:01.778 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.778 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:01.778 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:01.778 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:01.778 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:01.778 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:01.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.794+0000 7f48639f6640 1 Processor -- start 2024-03-29T11:25:01.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.795+0000 7f48639f6640 1 -- start start 2024-03-29T11:25:01.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.795+0000 7f48639f6640 10 monclient: init 2024-03-29T11:25:01.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.796+0000 7f48639f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.796+0000 7f48639f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:01.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.796+0000 7f48639f6640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0b0770 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:01.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.796+0000 7f48639f6640 10 monclient(hunting): picked mon.noname-a con 0x7f485c14a700 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:01.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.799+0000 7f4860f91640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f485c14a700 0x7f485c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47365/0 (socket says 172.21.15.64:47365) 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f48639f6640 10 monclient(hunting): picked mon.noname-c con 0x7f485c0ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f48639f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0aba70 con 0x7f485c14a700 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f48639f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0aecb0 con 0x7f485c0ac5a0 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f48639f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0af150 con 0x7f485c0b0770 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f48639f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f48639f6640 10 monclient(hunting): authenticate will time out at 1238.070459s 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f4861f93640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f485c0ac5a0 0x7f485c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:58037/0 (socket says 172.21.15.64:58037) 2024-03-29T11:25:01.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.800+0000 7f4861f93640 1 -- 172.21.15.64:0/4224735532 learned_addr learned my addr 172.21.15.64:0/4224735532 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:01.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296962378 0 0) 0x7f485c0af150 con 0x7f485c0b0770 2024-03-29T11:25:01.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:25:01.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003520 con 0x7f485c0b0770 2024-03-29T11:25:01.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486062360 0 0) 0x7f4848003520 con 0x7f485c0b0770 2024-03-29T11:25:01.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 >> v1:172.21.15.136:6789/0 conn(0x7f485c0ac5a0 legacy=0x7f485c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 >> v1:172.21.15.64:6789/0 conn(0x7f485c14a700 legacy=0x7f485c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 monclient: found mon.noname-b 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.802+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c0ac9d0 con 0x7f485c0b0770 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4854001a10 con 0x7f485c0b0770 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: dump: 2024-03-29T11:25:01.787 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4854001ca0 con 0x7f485c0b0770 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f48639f6640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f48639f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:01.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 1 -- 172.21.15.64:0/4224735532 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4854002030 con 0x7f485c0b0770 2024-03-29T11:25:01.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f4853fff640 10 monclient: dump: 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.790 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f48639f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.803+0000 7f48639f6640 4 set_mon_vals no callback set 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.804+0000 7f48639f6640 10 monclient: shutdown 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.804+0000 7f48639f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.804+0000 7f48639f6640 1 -- 172.21.15.64:0/4224735532 >> v1:172.21.15.64:6790/0 conn(0x7f485c0b0770 legacy=0x7f485c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.804+0000 7f48639f6640 1 -- 172.21.15.64:0/4224735532 shutdown_connections 2024-03-29T11:25:01.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.804+0000 7f48639f6640 1 -- 172.21.15.64:0/4224735532 >> 172.21.15.64:0/4224735532 conn(0x7f485c06b660 msgr2=0x7f485c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.806+0000 7f48639f6640 1 -- 172.21.15.64:0/4224735532 shutdown_connections 2024-03-29T11:25:01.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.808+0000 7f48639f6640 1 -- 172.21.15.64:0/4224735532 wait complete. 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.809+0000 7f48639f6640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.809+0000 7f48639f6640 10 monclient: monmap: 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:01.797 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.809+0000 7f48639f6640 1 librados: starting msgr at 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.809+0000 7f48639f6640 1 librados: starting objecter 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.809+0000 7f48639f6640 1 Processor -- start 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.810+0000 7f48639f6640 1 -- start start 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.810+0000 7f48639f6640 1 librados: setting wanted keys 2024-03-29T11:25:01.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.810+0000 7f48639f6640 1 librados: calling monclient init 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.810+0000 7f48639f6640 10 monclient: init 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient(hunting): picked mon.noname-a con 0x7f485c0ac5a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0b0770 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient(hunting): picked mon.noname-c con 0x7f485c14a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c11d3c0 con 0x7f485c0ac5a0 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c11d660 con 0x7f485c0b0770 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c11e290 con 0x7f485c14a700 2024-03-29T11:25:01.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.811+0000 7f48639f6640 10 monclient(hunting): authenticate will time out at 1238.081275s 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4860f91640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f485c0b0770 0x7f485c1e80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:58115/0 (socket says 172.21.15.64:58115) 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4860f91640 1 -- 172.21.15.64:0/3770938238 learned_addr learned my addr 172.21.15.64:0/3770938238 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 client.?.objecter ms_handle_connect 0x7f485c0b0770 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780112581 0 0) 0x7f485c11d660 con 0x7f485c0b0770 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840004eb0 con 0x7f485c0b0770 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2124148401 0 0) 0x7f4840004eb0 con 0x7f485c0b0770 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f485c11d660 con 0x7f485c0b0770 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f484c001890 con 0x7f485c0b0770 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:01.807 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 client.?.objecter ms_handle_connect 0x7f485c14a700 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 399002580 0 0) 0x7f485c11d660 con 0x7f485c0b0770 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.815+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 >> v1:172.21.15.64:6790/0 conn(0x7f485c14a700 legacy=0x7f485c11db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 >> v1:172.21.15.64:6789/0 conn(0x7f485c0ac5a0 legacy=0x7f485c1e79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 1 monclient: found mon.b 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c11e420 con 0x7f485c0b0770 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 10 monclient: _renew_subs 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:01.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c11e6f0 con 0x7f485c0b0770 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 10 monclient: _renew_subs 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c1efe90 con 0x7f485c0b0770 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f48639f6640 1 librados: init done 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f484c0019f0 con 0x7f485c0b0770 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f484c004a80 con 0x7f485c0b0770 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.816+0000 7f4851ffb640 10 monclient: dump: 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:01.810 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.818+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f484c01aa50 con 0x7f485c0b0770 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.818+0000 7f48639f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.818+0000 7f48639f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.818+0000 7f48639f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.818+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c11d850 con 0x7f485c0b0770 2024-03-29T11:25:01.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f484c084aa0 con 0x7f485c0b0770 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4851ffb640 10 client.4273.objecter ms_dispatch 0x7f485c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4851ffb640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4851ffb640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4851ffb640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4852ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.819+0000 7f4861792640 10 monclient: get_auth_request con 0x7f484006dc70 auth_method 0 2024-03-29T11:25:01.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.820+0000 7f4851ffb640 10 client.4273.objecter ms_handle_connect 0x7f484006dc70 2024-03-29T11:25:01.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.831+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f484c050380 con 0x7f485c0b0770 2024-03-29T11:25:01.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.831+0000 7f4851ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:01.831+0000 7f4851ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:02.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.161+0000 7f48639f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.161+0000 7f48639f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.161+0000 7f48639f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:02.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.161+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f485c11e880 con 0x7f485c0b0770 2024-03-29T11:25:02.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.164+0000 7f4851ffb640 1 -- 172.21.15.64:0/3770938238 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f484c0522b0 con 0x7f485c0b0770 2024-03-29T11:25:02.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.164+0000 7f4851ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.164+0000 7f4851ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:02.149 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:02.149 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:02.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.169+0000 7f48639f6640 10 librados: watch_flush enter 2024-03-29T11:25:02.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 10 librados: watch_flush exit 2024-03-29T11:25:02.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T11:25:02.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 10 client.4273.objecter successfully canceled tick 2024-03-29T11:25:02.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 >> v1:172.21.15.136:6816/2312274743 conn(0x7f484006dc70 legacy=0x7f4840070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 10 monclient: shutdown 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 >> v1:172.21.15.136:6789/0 conn(0x7f485c0b0770 legacy=0x7f485c1e80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 shutdown_connections 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 >> 172.21.15.64:0/3770938238 conn(0x7f485c06b660 msgr2=0x7f485c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.170+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 shutdown_connections 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.171+0000 7f48639f6640 1 -- 172.21.15.64:0/3770938238 wait complete. 2024-03-29T11:25:02.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:02.171+0000 7f48639f6640 1 librados: shutdown 2024-03-29T11:25:02.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:08.979 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:09.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.154+0000 7f0d29982640 10 monclient: get_monmap_and_config 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.154+0000 7f0d29982640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.154+0000 7f0d29982640 10 monclient: monmap: 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:09.156295+0000 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:09.156295+0000 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:09.139 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:09.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.155+0000 7f0d29982640 1 Processor -- start 2024-03-29T11:25:09.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.155+0000 7f0d29982640 1 -- start start 2024-03-29T11:25:09.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.155+0000 7f0d29982640 10 monclient: init 2024-03-29T11:25:09.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:09.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient(hunting): picked mon.noname-c con 0x7f0d240b0770 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:09.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient(hunting): picked mon.noname-b con 0x7f0d2414a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:09.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient(hunting): picked mon.noname-a con 0x7f0d240ac5a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:09.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240aba70 con 0x7f0d240ac5a0 2024-03-29T11:25:09.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240aecb0 con 0x7f0d240b0770 2024-03-29T11:25:09.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240af150 con 0x7f0d2414a700 2024-03-29T11:25:09.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.156+0000 7f0d29982640 10 monclient(hunting): authenticate will time out at 1245.427274s 2024-03-29T11:25:09.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.159+0000 7f0d22ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0d240b0770 0x7f0d24149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:45661/0 (socket says 172.21.15.64:45661) 2024-03-29T11:25:09.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.159+0000 7f0d22ffd640 1 -- 172.21.15.64:0/105111936 learned_addr learned my addr 172.21.15.64:0/105111936 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439344174 0 0) 0x7f0d240aecb0 con 0x7f0d240b0770 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c003520 con 0x7f0d240b0770 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106217161 0 0) 0x7f0d240af150 con 0x7f0d2414a700 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d240aecb0 con 0x7f0d2414a700 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497115614 0 0) 0x7f0d240aba70 con 0x7f0d240ac5a0 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d240af150 con 0x7f0d240ac5a0 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2339958917 0 0) 0x7f0d0c003520 con 0x7f0d240b0770 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 >> v1:172.21.15.64:6790/0 conn(0x7f0d2414a700 legacy=0x7f0d240abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 >> v1:172.21.15.64:6789/0 conn(0x7f0d240ac5a0 legacy=0x7f0d24151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 monclient: found mon.noname-c 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240ac9d0 con 0x7f0d240b0770 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0d140016b0 con 0x7f0d240b0770 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d21ffb640 10 monclient: dump: 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:09.151 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d29982640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.161+0000 7f0d29982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d14001810 con 0x7f0d240b0770 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d21ffb640 1 -- 172.21.15.64:0/105111936 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0d14001c20 con 0x7f0d240b0770 2024-03-29T11:25:09.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d21ffb640 10 monclient: dump: 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d29982640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:09.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.162+0000 7f0d29982640 4 set_mon_vals no callback set 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.163+0000 7f0d29982640 10 monclient: shutdown 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.163+0000 7f0d29982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.163+0000 7f0d29982640 1 -- 172.21.15.64:0/105111936 >> v1:172.21.15.136:6789/0 conn(0x7f0d240b0770 legacy=0x7f0d24149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.164+0000 7f0d29982640 1 -- 172.21.15.64:0/105111936 shutdown_connections 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.164+0000 7f0d29982640 1 -- 172.21.15.64:0/105111936 >> 172.21.15.64:0/105111936 conn(0x7f0d2406b660 msgr2=0x7f0d2406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.164+0000 7f0d29982640 1 -- 172.21.15.64:0/105111936 shutdown_connections 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.164+0000 7f0d29982640 1 -- 172.21.15.64:0/105111936 wait complete. 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 10 monclient: monmap: 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:09.154 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.155 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:09.155 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:09.155 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:09.155 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:09.155 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:09.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 1 librados: starting msgr at 2024-03-29T11:25:09.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 1 librados: starting objecter 2024-03-29T11:25:09.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 1 Processor -- start 2024-03-29T11:25:09.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 1 -- start start 2024-03-29T11:25:09.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 1 librados: setting wanted keys 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 1 librados: calling monclient init 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.165+0000 7f0d29982640 10 monclient: init 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient(hunting): picked mon.noname-b con 0x7f0d240ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient(hunting): picked mon.noname-a con 0x7f0d240b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient(hunting): picked mon.noname-c con 0x7f0d2414a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d241239d0 con 0x7f0d240b0770 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24128140 con 0x7f0d240ac5a0 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24126070 con 0x7f0d2414a700 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.166+0000 7f0d29982640 10 monclient(hunting): authenticate will time out at 1245.437580s 2024-03-29T11:25:09.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d237fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0d2414a700 0x7f0d241e6fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46252/0 (socket says 172.21.15.64:46252) 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d237fe640 1 -- 172.21.15.64:0/3724148490 learned_addr learned my addr 172.21.15.64:0/3724148490 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 10 client.?.objecter ms_handle_connect 0x7f0d2414a700 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300359703 0 0) 0x7f0d24126070 con 0x7f0d2414a700 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.168+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10004eb0 con 0x7f0d2414a700 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 client.?.objecter ms_handle_connect 0x7f0d240b0770 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903529590 0 0) 0x7f0d10004eb0 con 0x7f0d2414a700 2024-03-29T11:25:09.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d24126070 con 0x7f0d2414a700 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0d1c001d40 con 0x7f0d2414a700 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:09.161 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943660702 0 0) 0x7f0d241239d0 con 0x7f0d240b0770 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10004eb0 con 0x7f0d240b0770 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.171+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1323305032 0 0) 0x7f0d24126070 con 0x7f0d2414a700 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 >> v1:172.21.15.136:6789/0 conn(0x7f0d240ac5a0 legacy=0x7f0d24144d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:09.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 >> v1:172.21.15.64:6789/0 conn(0x7f0d240b0770 legacy=0x7f0d24126870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 1 monclient: found mon.c 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d24126350 con 0x7f0d2414a700 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 10 monclient: _renew_subs 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d24125dc0 con 0x7f0d2414a700 2024-03-29T11:25:09.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:09.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 10 monclient: _renew_subs 2024-03-29T11:25:09.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:09.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d241dfab0 con 0x7f0d2414a700 2024-03-29T11:25:09.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.172+0000 7f0d29982640 1 librados: init done 2024-03-29T11:25:09.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.175+0000 7f0d29982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.175+0000 7f0d29982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.175+0000 7f0d29982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.175+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d240ab560 con 0x7f0d2414a700 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d1c001ea0 con 0x7f0d2414a700 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d217fa640 4 set_mon_vals no callback set 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0d1c003de0 con 0x7f0d2414a700 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:09.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.176+0000 7f0d0bfff640 10 monclient: dump: 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:09.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.178+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f0d1c003fc0 con 0x7f0d2414a700 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.178+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f0d1c084e60 con 0x7f0d2414a700 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.179+0000 7f0d22ffd640 10 monclient: get_auth_request con 0x7f0d1006db20 auth_method 0 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.179+0000 7f0d0bfff640 10 client.4280.objecter ms_dispatch 0x7f0d24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.179+0000 7f0d0bfff640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.179+0000 7f0d0bfff640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.179+0000 7f0d0bfff640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T11:25:09.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.180+0000 7f0d0bfff640 10 client.4280.objecter ms_handle_connect 0x7f0d1006db20 2024-03-29T11:25:09.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.187+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d1c050740 con 0x7f0d2414a700 2024-03-29T11:25:09.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.187+0000 7f0d0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.187+0000 7f0d0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:09.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.519+0000 7f0d29982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.519+0000 7f0d29982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.519+0000 7f0d29982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:09.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.519+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d240b0b50 con 0x7f0d2414a700 2024-03-29T11:25:09.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.521+0000 7f0d0bfff640 1 -- 172.21.15.64:0/3724148490 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f0d1c052670 con 0x7f0d2414a700 2024-03-29T11:25:09.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.521+0000 7f0d0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.521+0000 7f0d0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:09.505 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:09.506 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 10 librados: watch_flush enter 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 10 librados: watch_flush exit 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 10 client.4280.objecter successfully canceled tick 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 >> v1:172.21.15.136:6816/2312274743 conn(0x7f0d1006db20 legacy=0x7f0d1006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 10 monclient: shutdown 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.524+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 >> v1:172.21.15.64:6790/0 conn(0x7f0d2414a700 legacy=0x7f0d241e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.525+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 shutdown_connections 2024-03-29T11:25:09.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.525+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 >> 172.21.15.64:0/3724148490 conn(0x7f0d2406b660 msgr2=0x7f0d240abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.527+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 shutdown_connections 2024-03-29T11:25:09.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.527+0000 7f0d29982640 1 -- 172.21.15.64:0/3724148490 wait complete. 2024-03-29T11:25:09.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:09.527+0000 7f0d29982640 1 librados: shutdown 2024-03-29T11:25:09.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:16.335 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:16.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.506+0000 7fca3c379640 10 monclient: get_monmap_and_config 2024-03-29T11:25:16.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.506+0000 7fca3c379640 10 monclient: build_initial_monmap 2024-03-29T11:25:16.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.506+0000 7fca3c379640 10 monclient: monmap: 2024-03-29T11:25:16.490 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:16.490 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:16.490 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:16.507678+0000 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:16.507678+0000 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:16.491 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.507+0000 7fca3c379640 1 Processor -- start 2024-03-29T11:25:16.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.507+0000 7fca3c379640 1 -- start start 2024-03-29T11:25:16.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.507+0000 7fca3c379640 10 monclient: init 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.508+0000 7fca3c379640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.508+0000 7fca3c379640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 10 monclient(hunting): picked mon.noname-b con 0x7fca34071de0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 10 monclient(hunting): picked mon.noname-c con 0x7fca340721c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 10 monclient(hunting): picked mon.noname-a con 0x7fca34145ae0 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca34145650 con 0x7fca34145ae0 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca340ae260 con 0x7fca340721c0 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca341420d0 con 0x7fca34071de0 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:16.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3c379640 10 monclient(hunting): authenticate will time out at 1252.779126s 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3a115640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fca34071de0 0x7fca340aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41888/0 (socket says 172.21.15.64:41888) 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.509+0000 7fca3a115640 1 -- 172.21.15.64:0/1244817117 learned_addr learned my addr 172.21.15.64:0/1244817117 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66119032 0 0) 0x7fca340ae260 con 0x7fca340721c0 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca20003520 con 0x7fca340721c0 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135476499 0 0) 0x7fca341420d0 con 0x7fca34071de0 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca340ae260 con 0x7fca34071de0 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196840567 0 0) 0x7fca20003520 con 0x7fca340721c0 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 >> v1:172.21.15.64:6790/0 conn(0x7fca34071de0 legacy=0x7fca340aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 >> v1:172.21.15.64:6789/0 conn(0x7fca34145ae0 legacy=0x7fca34145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: _finish_hunting 0 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 monclient: found mon.noname-c 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca340b0590 con 0x7fca340721c0 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: _finish_auth 0 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: _check_auth_tickets 2024-03-29T11:25:16.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fca2c001a10 con 0x7fca340721c0 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca39113640 10 monclient: dump: 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:16.498 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca3c379640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.510+0000 7fca3c379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca2c001b70 con 0x7fca340721c0 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca39113640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca39113640 1 -- 172.21.15.64:0/1244817117 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fca2c001f00 con 0x7fca340721c0 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca39113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca39113640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca39113640 10 monclient: dump: 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:16.499 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca3c379640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:16.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.511+0000 7fca3c379640 4 set_mon_vals no callback set 2024-03-29T11:25:16.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 10 monclient: shutdown 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 1 -- 172.21.15.64:0/1244817117 >> v1:172.21.15.136:6789/0 conn(0x7fca340721c0 legacy=0x7fca3413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 1 -- 172.21.15.64:0/1244817117 shutdown_connections 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 1 -- 172.21.15.64:0/1244817117 >> 172.21.15.64:0/1244817117 conn(0x7fca3406b660 msgr2=0x7fca3406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 1 -- 172.21.15.64:0/1244817117 shutdown_connections 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.512+0000 7fca3c379640 1 -- 172.21.15.64:0/1244817117 wait complete. 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.513+0000 7fca3c379640 10 monclient: build_initial_monmap 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.513+0000 7fca3c379640 10 monclient: monmap: 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:16.508 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 1 librados: starting msgr at 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 1 librados: starting objecter 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 1 Processor -- start 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 1 -- start start 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 1 librados: setting wanted keys 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 1 librados: calling monclient init 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.514+0000 7fca3c379640 10 monclient: init 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.515+0000 7fca3c379640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.515+0000 7fca3c379640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:16.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.515+0000 7fca3c379640 10 monclient(hunting): picked mon.noname-c con 0x7fca34071de0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.515+0000 7fca3a115640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fca34071de0 0x7fca34138ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42054/0 (socket says 172.21.15.64:42054) 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.515+0000 7fca3c379640 10 monclient(hunting): picked mon.noname-b con 0x7fca34145ae0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3c379640 10 monclient(hunting): picked mon.noname-a con 0x7fca34137fb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3a115640 1 -- 172.21.15.64:0/1670957822 learned_addr learned my addr 172.21.15.64:0/1670957822 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca341339e0 con 0x7fca34137fb0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca341dee40 con 0x7fca34145ae0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3406e130 con 0x7fca34071de0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3c379640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca3c379640 10 monclient(hunting): authenticate will time out at 1252.786013s 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 client.?.objecter ms_handle_connect 0x7fca34145ae0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 client.?.objecter ms_handle_connect 0x7fca34071de0 2024-03-29T11:25:16.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444583518 0 0) 0x7fca341dee40 con 0x7fca34145ae0 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca18004eb0 con 0x7fca34145ae0 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927013592 0 0) 0x7fca3406e130 con 0x7fca34071de0 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.516+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca341dee40 con 0x7fca34071de0 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 client.?.objecter ms_handle_connect 0x7fca34137fb0 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 300735713 0 0) 0x7fca18004eb0 con 0x7fca34145ae0 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca3406e130 con 0x7fca34145ae0 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970851411 0 0) 0x7fca341dee40 con 0x7fca34071de0 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca18004eb0 con 0x7fca34071de0 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fca2c001e80 con 0x7fca34145ae0 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): dump: 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.512 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fca30001810 con 0x7fca34071de0 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient(hunting): dump: 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:16.513 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1152054766 0 0) 0x7fca3406e130 con 0x7fca34145ae0 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 >> v1:172.21.15.64:6790/0 conn(0x7fca34071de0 legacy=0x7fca34138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 >> v1:172.21.15.64:6789/0 conn(0x7fca34137fb0 legacy=0x7fca341332d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 monclient: found mon.b 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:16.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3406e410 con 0x7fca34145ae0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient: _finish_auth 0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 10 monclient: _renew_subs 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3406e6f0 con 0x7fca34145ae0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 10 monclient: _renew_subs 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.517+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca340aca10 con 0x7fca34145ae0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.518+0000 7fca3c379640 1 librados: init done 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.519+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca2c003c20 con 0x7fca34145ae0 2024-03-29T11:25:16.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.519+0000 7fca2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.519+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fca2c003f30 con 0x7fca34145ae0 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.519+0000 7fca2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.519+0000 7fca2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.520+0000 7fca38912640 4 set_mon_vals no callback set 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.520+0000 7fca3c379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.519+0000 7fca2affd640 10 monclient: dump: 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:16.516 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.521+0000 7fca3c379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.521+0000 7fca3c379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.521+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c005080 con 0x7fca34145ae0 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.521+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fca2c019f00 con 0x7fca34145ae0 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.523+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fca2c083d10 con 0x7fca34145ae0 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.523+0000 7fca2affd640 10 client.4282.objecter ms_dispatch 0x7fca34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.523+0000 7fca2affd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.523+0000 7fca2affd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.523+0000 7fca2affd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:25:16.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.523+0000 7fca3a115640 10 monclient: get_auth_request con 0x7fca1806df50 auth_method 0 2024-03-29T11:25:16.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.524+0000 7fca2affd640 10 client.4282.objecter ms_handle_connect 0x7fca1806df50 2024-03-29T11:25:16.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.528+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca2c04f5f0 con 0x7fca34145ae0 2024-03-29T11:25:16.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.528+0000 7fca2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.528+0000 7fca2affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:16.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.930+0000 7fca3c379640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.930+0000 7fca3c379640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.930+0000 7fca3c379640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:16.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.930+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca1c005330 con 0x7fca34145ae0 2024-03-29T11:25:16.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.931+0000 7fca2affd640 1 -- 172.21.15.64:0/1670957822 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fca2c051520 con 0x7fca34145ae0 2024-03-29T11:25:16.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.931+0000 7fca2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.931+0000 7fca2affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:16.915 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:16.915 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.934+0000 7fca3c379640 10 librados: watch_flush enter 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.934+0000 7fca3c379640 10 librados: watch_flush exit 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.934+0000 7fca3c379640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 >> v1:172.21.15.136:6816/2312274743 conn(0x7fca1806df50 legacy=0x7fca18070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 10 monclient: shutdown 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 >> v1:172.21.15.136:6789/0 conn(0x7fca34145ae0 legacy=0x7fca34132bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 shutdown_connections 2024-03-29T11:25:16.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 >> 172.21.15.64:0/1670957822 conn(0x7fca3406b660 msgr2=0x7fca340a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 shutdown_connections 2024-03-29T11:25:16.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 -- 172.21.15.64:0/1670957822 wait complete. 2024-03-29T11:25:16.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:16.935+0000 7fca3c379640 1 librados: shutdown 2024-03-29T11:25:16.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:23.741 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.891+0000 7f7c208c1640 10 monclient: get_monmap_and_config 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.891+0000 7f7c208c1640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.891+0000 7f7c208c1640 10 monclient: monmap: 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:23.892241+0000 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:23.892241+0000 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:23.875 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:23.876 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:23.876 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:23.876 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.892+0000 7f7c208c1640 1 Processor -- start 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.892+0000 7f7c208c1640 1 -- start start 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.892+0000 7f7c208c1640 10 monclient: init 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1c1457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c1c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c0aaef0 con 0x7f7c1c0a9d70 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c145bd0 con 0x7f7c1c1457f0 2024-03-29T11:25:23.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c145eb0 con 0x7f7c1c13f650 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c208c1640 10 monclient(hunting): authenticate will time out at 1260.163175s 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c19d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7c1c13f650 0x7f7c1c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49814/0 (socket says 172.21.15.64:49814) 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.893+0000 7f7c19d9b640 1 -- 172.21.15.64:0/602759994 learned_addr learned my addr 172.21.15.64:0/602759994 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.894+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854079211 0 0) 0x7f7c1c145eb0 con 0x7f7c1c13f650 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.894+0000 7f7c1959a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.894+0000 7f7c1959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.894+0000 7f7c1959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.894+0000 7f7c1959a640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:25:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.894+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c04003520 con 0x7f7c1c13f650 2024-03-29T11:25:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.895+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027096123 0 0) 0x7f7c04003520 con 0x7f7c1c13f650 2024-03-29T11:25:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.895+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 >> v1:172.21.15.136:6789/0 conn(0x7f7c1c1457f0 legacy=0x7f7c1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.895+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 >> v1:172.21.15.64:6789/0 conn(0x7f7c1c0a9d70 legacy=0x7f7c1c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 1 monclient: found mon.noname-b 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c146340 con 0x7f7c1c13f650 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7c10001690 con 0x7f7c1c13f650 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: dump: 2024-03-29T11:25:23.881 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c208c1640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c208c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c10001860 con 0x7f7c1c13f650 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 1 -- 172.21.15.64:0/602759994 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7c10001cb0 con 0x7f7c1c13f650 2024-03-29T11:25:23.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c1959a640 10 monclient: dump: 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c208c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.896+0000 7f7c208c1640 4 set_mon_vals no callback set 2024-03-29T11:25:23.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.897+0000 7f7c208c1640 10 monclient: shutdown 2024-03-29T11:25:23.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.897+0000 7f7c208c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.897+0000 7f7c208c1640 1 -- 172.21.15.64:0/602759994 >> v1:172.21.15.64:6790/0 conn(0x7f7c1c13f650 legacy=0x7f7c1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.897+0000 7f7c208c1640 1 -- 172.21.15.64:0/602759994 shutdown_connections 2024-03-29T11:25:23.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.897+0000 7f7c208c1640 1 -- 172.21.15.64:0/602759994 >> 172.21.15.64:0/602759994 conn(0x7f7c1c066a00 msgr2=0x7f7c1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.900+0000 7f7c208c1640 1 -- 172.21.15.64:0/602759994 shutdown_connections 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.900+0000 7f7c208c1640 1 -- 172.21.15.64:0/602759994 wait complete. 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.901+0000 7f7c208c1640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.901+0000 7f7c208c1640 10 monclient: monmap: 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:23.885 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.886 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:23.886 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:23.886 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:23.886 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:23.886 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 1 librados: starting msgr at 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 1 librados: starting objecter 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 1 Processor -- start 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 1 -- start start 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 1 librados: setting wanted keys 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 1 librados: calling monclient init 2024-03-29T11:25:23.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.902+0000 7f7c208c1640 10 monclient: init 2024-03-29T11:25:23.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.904+0000 7f7c208c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.904+0000 7f7c208c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:23.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.904+0000 7f7c208c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c1c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:23.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.904+0000 7f7c1a59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7c1c0a9d70 0x7f7c1c0703d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:48032/0 (socket says 172.21.15.64:48032) 2024-03-29T11:25:23.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:23.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1c1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:23.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c1ad9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7c1c1457f0 0x7f7c1c1df500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58394/0 (socket says 172.21.15.64:58394) 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c1ad9d640 1 -- 172.21.15.64:0/1395385713 learned_addr learned my addr 172.21.15.64:0/1395385713 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c13abe0 con 0x7f7c1c13f650 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c1dfc10 con 0x7f7c1c0a9d70 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c0affd640 10 client.?.objecter ms_handle_connect 0x7f7c1c1457f0 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c071a40 con 0x7f7c1c1457f0 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c208c1640 10 monclient(hunting): authenticate will time out at 1260.175623s 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904037647 0 0) 0x7f7c1c071a40 con 0x7f7c1c1457f0 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.905+0000 7f7c0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bfc004eb0 con 0x7f7c1c1457f0 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718304386 0 0) 0x7f7bfc004eb0 con 0x7f7c1c1457f0 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c1c071a40 con 0x7f7c1c1457f0 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7c0c004000 con 0x7f7c1c1457f0 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): dump: 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:23.894 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.906+0000 7f7c0affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 client.?.objecter ms_handle_connect 0x7f7c1c0a9d70 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1518824987 0 0) 0x7f7c1c071a40 con 0x7f7c1c1457f0 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 >> v1:172.21.15.136:6789/0 conn(0x7f7c1c0a9d70 legacy=0x7f7c1c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 >> v1:172.21.15.64:6789/0 conn(0x7f7c1c13f650 legacy=0x7f7c1c1dedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 1 monclient: found mon.c 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c071d20 con 0x7f7c1c1457f0 2024-03-29T11:25:23.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 client.?.objecter ms_handle_connect 0x7f7c1c13f650 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c208c1640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.907+0000 7f7c208c1640 10 monclient: _renew_subs 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c072000 con 0x7f7c1c1457f0 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c0c004450 con 0x7f7c1c1457f0 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7c0c001180 con 0x7f7c1c1457f0 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c0affd640 10 monclient: dump: 2024-03-29T11:25:23.896 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c18d99640 4 set_mon_vals no callback set 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 10 monclient: _renew_subs 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c06fe80 con 0x7f7c1c1457f0 2024-03-29T11:25:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.908+0000 7f7c208c1640 1 librados: init done 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.910+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f7c0c0012e0 con 0x7f7c1c1457f0 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.910+0000 7f7c208c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.910+0000 7f7c208c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.910+0000 7f7c208c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.910+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c075f10 con 0x7f7c1c1457f0 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c1a59c640 10 monclient: get_auth_request con 0x7f7bfc06dd50 auth_method 0 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f7c0c084ac0 con 0x7f7c1c1457f0 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c0affd640 10 client.4292.objecter ms_dispatch 0x7f7c1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c0affd640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c0affd640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c0affd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:25:23.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.911+0000 7f7c0affd640 10 client.4292.objecter ms_handle_connect 0x7f7bfc06dd50 2024-03-29T11:25:23.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.918+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c0c0503a0 con 0x7f7c1c1457f0 2024-03-29T11:25:23.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.919+0000 7f7c0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:23.919+0000 7f7c0affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.240+0000 7f7c208c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.240+0000 7f7c208c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.240+0000 7f7c208c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.240+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c1c0b3680 con 0x7f7c1c1457f0 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.241+0000 7f7c0affd640 1 -- 172.21.15.64:0/1395385713 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f7c0c0522d0 con 0x7f7c1c1457f0 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.241+0000 7f7c0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.225 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:24.226 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:24.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.241+0000 7f7c0affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.243+0000 7f7c208c1640 10 librados: watch_flush enter 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 10 librados: watch_flush exit 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 >> v1:172.21.15.136:6816/2312274743 conn(0x7f7bfc06dd50 legacy=0x7f7bfc070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 10 monclient: shutdown 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 >> v1:172.21.15.64:6790/0 conn(0x7f7c1c1457f0 legacy=0x7f7c1c1df500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 shutdown_connections 2024-03-29T11:25:24.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 >> 172.21.15.64:0/1395385713 conn(0x7f7c1c066a00 msgr2=0x7f7c1c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:24.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 shutdown_connections 2024-03-29T11:25:24.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 -- 172.21.15.64:0/1395385713 wait complete. 2024-03-29T11:25:24.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:24.244+0000 7f7c208c1640 1 librados: shutdown 2024-03-29T11:25:24.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:31.052 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:31.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.217+0000 7fee5359e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.217+0000 7fee5359e640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.218+0000 7fee5359e640 10 monclient: monmap: 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:31.219311+0000 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:31.219311+0000 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:31.203 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:31.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.220+0000 7fee5359e640 1 Processor -- start 2024-03-29T11:25:31.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.220+0000 7fee5359e640 1 -- start start 2024-03-29T11:25:31.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.220+0000 7fee5359e640 10 monclient: init 2024-03-29T11:25:31.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.221+0000 7fee5359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.221+0000 7fee5359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:31.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.221+0000 7fee5359e640 10 monclient(hunting): picked mon.noname-a con 0x7fee4c038d00 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 10 monclient(hunting): picked mon.noname-b con 0x7fee4c0c49d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 10 monclient(hunting): picked mon.noname-c con 0x7fee4c0c7820 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c02f790 con 0x7fee4c038d00 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0307c0 con 0x7fee4c0c7820 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c02f270 con 0x7fee4c0c49d0 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.224+0000 7fee5359e640 10 monclient(hunting): authenticate will time out at 1267.494856s 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.225+0000 7fee5259c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fee4c038d00 0x7fee4c02e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34773/0 (socket says 172.21.15.64:34773) 2024-03-29T11:25:31.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.225+0000 7fee5259c640 1 -- 172.21.15.64:0/897367210 learned_addr learned my addr 172.21.15.64:0/897367210 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942332708 0 0) 0x7fee4c02f270 con 0x7fee4c0c49d0 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c003520 con 0x7fee4c0c49d0 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837027240 0 0) 0x7fee4c02f790 con 0x7fee4c038d00 2024-03-29T11:25:31.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:25:31.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.227+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c02f270 con 0x7fee4c038d00 2024-03-29T11:25:31.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394230603 0 0) 0x7fee4c0307c0 con 0x7fee4c0c7820 2024-03-29T11:25:31.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c02f790 con 0x7fee4c0c7820 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2433006478 0 0) 0x7fee3c003520 con 0x7fee4c0c49d0 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 >> v1:172.21.15.136:6789/0 conn(0x7fee4c0c7820 legacy=0x7fee4c0c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 >> v1:172.21.15.64:6789/0 conn(0x7fee4c038d00 legacy=0x7fee4c02e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 monclient: found mon.noname-b 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0c8350 con 0x7fee4c0c49d0 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fee44001610 con 0x7fee4c0c49d0 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5159a640 10 monclient: dump: 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:31.214 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:31.215 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:31.215 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:31.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5359e640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:25:31.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.228+0000 7fee5359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:31.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee44001770 con 0x7fee4c0c49d0 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5159a640 1 -- 172.21.15.64:0/897367210 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fee44001bc0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5159a640 10 monclient: dump: 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:31.216 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.229+0000 7fee5359e640 4 set_mon_vals no callback set 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.230+0000 7fee5359e640 10 monclient: shutdown 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.230+0000 7fee5359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.230+0000 7fee5359e640 1 -- 172.21.15.64:0/897367210 >> v1:172.21.15.64:6790/0 conn(0x7fee4c0c49d0 legacy=0x7fee4c0c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.231+0000 7fee5359e640 1 -- 172.21.15.64:0/897367210 shutdown_connections 2024-03-29T11:25:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.231+0000 7fee5359e640 1 -- 172.21.15.64:0/897367210 >> 172.21.15.64:0/897367210 conn(0x7fee4c005c60 msgr2=0x7fee4c006070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.231+0000 7fee5359e640 1 -- 172.21.15.64:0/897367210 shutdown_connections 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.231+0000 7fee5359e640 1 -- 172.21.15.64:0/897367210 wait complete. 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.232+0000 7fee5359e640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.232+0000 7fee5359e640 10 monclient: monmap: 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:31.218 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.233+0000 7fee5359e640 1 librados: starting msgr at 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.233+0000 7fee5359e640 1 librados: starting objecter 2024-03-29T11:25:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.233+0000 7fee5359e640 1 Processor -- start 2024-03-29T11:25:31.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.236+0000 7fee5359e640 1 -- start start 2024-03-29T11:25:31.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.236+0000 7fee5359e640 1 librados: setting wanted keys 2024-03-29T11:25:31.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.236+0000 7fee5359e640 1 librados: calling monclient init 2024-03-29T11:25:31.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.236+0000 7fee5359e640 10 monclient: init 2024-03-29T11:25:31.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.237+0000 7fee5359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.238+0000 7fee5359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.238+0000 7fee5359e640 10 monclient(hunting): picked mon.noname-c con 0x7fee4c038d00 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.238+0000 7fee5359e640 10 monclient(hunting): picked mon.noname-b con 0x7fee4c0c49d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.238+0000 7fee5359e640 10 monclient(hunting): picked mon.noname-a con 0x7fee4c0c7820 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.238+0000 7fee5359e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0c7100 con 0x7fee4c0c7820 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee5359e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0b7c80 con 0x7fee4c0c49d0 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee51d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fee4c0c49d0 0x7fee4c0b6e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:57866/0 (socket says 172.21.15.64:57866) 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee51d9b640 1 -- 172.21.15.64:0/3199733086 learned_addr learned my addr 172.21.15.64:0/3199733086 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee5359e640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c0bcfd0 con 0x7fee4c038d00 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee5359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 10 client.?.objecter ms_handle_connect 0x7fee4c0c49d0 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee5359e640 10 monclient(hunting): authenticate will time out at 1267.509180s 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581610079 0 0) 0x7fee4c0b7c80 con 0x7fee4c0c49d0 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.239+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee34004eb0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296679547 0 0) 0x7fee34004eb0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee4c0b7c80 con 0x7fee4c0c49d0 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fee44001610 con 0x7fee4c0c49d0 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:31.225 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:31.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4203952343 0 0) 0x7fee4c0b7c80 con 0x7fee4c0c49d0 2024-03-29T11:25:31.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.240+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 >> v1:172.21.15.64:6790/0 conn(0x7fee4c038d00 legacy=0x7fee4c0bb460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 >> v1:172.21.15.64:6789/0 conn(0x7fee4c0c7820 legacy=0x7fee4c0b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 1 monclient: found mon.b 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0c73e0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 10 client.?.objecter ms_handle_connect 0x7fee4c0c7820 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.242+0000 7fee5359e640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 10 monclient: _renew_subs 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c0c5620 con 0x7fee4c0c49d0 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee440038a0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fee44003cf0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee42ffd640 10 monclient: dump: 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:31.229 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 10 monclient: _renew_subs 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c0c0f60 con 0x7fee4c0c49d0 2024-03-29T11:25:31.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.243+0000 7fee5359e640 1 librados: init done 2024-03-29T11:25:31.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.244+0000 7fee50d99640 4 set_mon_vals no callback set 2024-03-29T11:25:31.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.244+0000 7fee40ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.245+0000 7fee40ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.245+0000 7fee40ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:31.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.245+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee38005080 con 0x7fee4c0c49d0 2024-03-29T11:25:31.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.251+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fee44003ed0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.251+0000 7fee5259c640 10 monclient: get_auth_request con 0x7fee3406dba0 auth_method 0 2024-03-29T11:25:31.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.251+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fee44084ca0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.251+0000 7fee42ffd640 10 client.4291.objecter ms_dispatch 0x7fee54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:31.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.251+0000 7fee42ffd640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:31.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.251+0000 7fee42ffd640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:31.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.252+0000 7fee42ffd640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:25:31.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.252+0000 7fee42ffd640 10 client.4291.objecter ms_handle_connect 0x7fee3406dba0 2024-03-29T11:25:31.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.255+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee44050580 con 0x7fee4c0c49d0 2024-03-29T11:25:31.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.255+0000 7fee42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.255+0000 7fee42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:31.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.580+0000 7fee40ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.580+0000 7fee40ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.580+0000 7fee40ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:31.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.580+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee38005330 con 0x7fee4c0c49d0 2024-03-29T11:25:31.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.581+0000 7fee42ffd640 1 -- 172.21.15.64:0/3199733086 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fee440524b0 con 0x7fee4c0c49d0 2024-03-29T11:25:31.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.581+0000 7fee42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.581+0000 7fee42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:31.566 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:31.566 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:31.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.583+0000 7fee40ff9640 10 librados: watch_flush enter 2024-03-29T11:25:31.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.583+0000 7fee40ff9640 10 librados: watch_flush exit 2024-03-29T11:25:31.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.583+0000 7fee40ff9640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:25:31.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.583+0000 7fee40ff9640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:25:31.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.583+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 >> v1:172.21.15.136:6816/2312274743 conn(0x7fee3406dba0 legacy=0x7fee34070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.583+0000 7fee40ff9640 10 monclient: shutdown 2024-03-29T11:25:31.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.584+0000 7fee40ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.584+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 >> v1:172.21.15.136:6789/0 conn(0x7fee4c0c49d0 legacy=0x7fee4c0b6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.584+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 shutdown_connections 2024-03-29T11:25:31.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.584+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 >> 172.21.15.64:0/3199733086 conn(0x7fee4c005c60 msgr2=0x7fee4c02e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.584+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 shutdown_connections 2024-03-29T11:25:31.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.585+0000 7fee40ff9640 1 -- 172.21.15.64:0/3199733086 wait complete. 2024-03-29T11:25:31.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:31.585+0000 7fee40ff9640 1 librados: shutdown 2024-03-29T11:25:31.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:38.392 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.551+0000 7f9d2e447640 10 monclient: get_monmap_and_config 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.551+0000 7f9d2e447640 10 monclient: build_initial_monmap 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.551+0000 7f9d2e447640 10 monclient: monmap: 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:38.552726+0000 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:38.552726+0000 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:38.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.554+0000 7f9d2e447640 1 Processor -- start 2024-03-29T11:25:38.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.554+0000 7f9d2e447640 1 -- start start 2024-03-29T11:25:38.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.554+0000 7f9d2e447640 10 monclient: init 2024-03-29T11:25:38.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient(hunting): picked mon.noname-b con 0x7f9d2806d9d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient(hunting): picked mon.noname-c con 0x7f9d2813e820 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient(hunting): picked mon.noname-a con 0x7f9d280af1d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d280aed10 con 0x7f9d280af1d0 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d280afd50 con 0x7f9d2813e820 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d280ab490 con 0x7f9d2806d9d0 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:38.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.555+0000 7f9d2e447640 10 monclient(hunting): authenticate will time out at 1274.825579s 2024-03-29T11:25:38.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d277fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9d2813e820 0x7f9d2813ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:51366/0 (socket says 172.21.15.64:51366) 2024-03-29T11:25:38.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d277fe640 1 -- 172.21.15.64:0/3544949400 learned_addr learned my addr 172.21.15.64:0/3544949400 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241063062 0 0) 0x7f9d280aed10 con 0x7f9d280af1d0 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d26ffd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.558+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d10003520 con 0x7f9d280af1d0 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272786159 0 0) 0x7f9d280ab490 con 0x7f9d2806d9d0 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d280aed10 con 0x7f9d2806d9d0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3024555057 0 0) 0x7f9d10003520 con 0x7f9d280af1d0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 >> v1:172.21.15.64:6790/0 conn(0x7f9d2806d9d0 legacy=0x7f9d280b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 >> v1:172.21.15.136:6789/0 conn(0x7f9d2813e820 legacy=0x7f9d2813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 monclient: found mon.noname-a 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d280ab770 con 0x7f9d280af1d0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9d20001a10 con 0x7f9d280af1d0 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.559+0000 7f9d26ffd640 10 monclient: dump: 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.560+0000 7f9d2e447640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.560+0000 7f9d2e447640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.561+0000 7f9d26ffd640 1 -- 172.21.15.64:0/3544949400 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d20001b70 con 0x7f9d280af1d0 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.561+0000 7f9d26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.561+0000 7f9d2e447640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.561+0000 7f9d2e447640 4 set_mon_vals no callback set 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.562+0000 7f9d2e447640 10 monclient: shutdown 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.562+0000 7f9d2e447640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.562+0000 7f9d2e447640 1 -- 172.21.15.64:0/3544949400 >> v1:172.21.15.64:6789/0 conn(0x7f9d280af1d0 legacy=0x7f9d280af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.562+0000 7f9d2e447640 1 -- 172.21.15.64:0/3544949400 shutdown_connections 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.562+0000 7f9d2e447640 1 -- 172.21.15.64:0/3544949400 >> 172.21.15.64:0/3544949400 conn(0x7f9d2806b660 msgr2=0x7f9d2806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.562+0000 7f9d2e447640 1 -- 172.21.15.64:0/3544949400 shutdown_connections 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.563+0000 7f9d2e447640 1 -- 172.21.15.64:0/3544949400 wait complete. 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.563+0000 7f9d2e447640 10 monclient: build_initial_monmap 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.563+0000 7f9d2e447640 10 monclient: monmap: 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 1 librados: starting msgr at 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 1 librados: starting objecter 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 1 Processor -- start 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 1 -- start start 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 1 librados: setting wanted keys 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 1 librados: calling monclient init 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.564+0000 7f9d2e447640 10 monclient: init 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.565+0000 7f9d2e447640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.565+0000 7f9d2e447640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.565+0000 7f9d2e447640 10 monclient(hunting): picked mon.noname-b con 0x7f9d2806d9d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.566+0000 7f9d2e447640 10 monclient(hunting): picked mon.noname-c con 0x7f9d280af1d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.567+0000 7f9d2e447640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2813e820 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.567+0000 7f9d2e447640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2806eb10 con 0x7f9d2813e820 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.567+0000 7f9d2e447640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2806edf0 con 0x7f9d2806d9d0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.567+0000 7f9d2e447640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d281d6ee0 con 0x7f9d280af1d0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.567+0000 7f9d2e447640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.567+0000 7f9d2e447640 10 monclient(hunting): authenticate will time out at 1274.836934s 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.568+0000 7f9d27fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9d2806d9d0 0x7f9d281346a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:52120/0 (socket says 172.21.15.64:52120) 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.568+0000 7f9d27fff640 1 -- 172.21.15.64:0/3284937956 learned_addr learned my addr 172.21.15.64:0/3284937956 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.569+0000 7f9d24ff9640 10 client.?.objecter ms_handle_connect 0x7f9d280af1d0 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.569+0000 7f9d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.569+0000 7f9d24ff9640 10 client.?.objecter ms_handle_connect 0x7f9d2806d9d0 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.569+0000 7f9d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048971461 0 0) 0x7f9d281d6ee0 con 0x7f9d280af1d0 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d14004eb0 con 0x7f9d280af1d0 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 client.?.objecter ms_handle_connect 0x7f9d2813e820 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649706372 0 0) 0x7f9d2806edf0 con 0x7f9d2806d9d0 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d281d6ee0 con 0x7f9d2806d9d0 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 920291651 0 0) 0x7f9d14004eb0 con 0x7f9d280af1d0 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d2806edf0 con 0x7f9d280af1d0 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.570+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597807690 0 0) 0x7f9d2806eb10 con 0x7f9d2813e820 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d14004eb0 con 0x7f9d2813e820 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273309764 0 0) 0x7f9d281d6ee0 con 0x7f9d2806d9d0 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d2806eb10 con 0x7f9d2806d9d0 2024-03-29T11:25:38.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9d1c001930 con 0x7f9d280af1d0 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:38.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9d18001810 con 0x7f9d2806d9d0 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:38.564 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1287681785 0 0) 0x7f9d2806edf0 con 0x7f9d280af1d0 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 >> v1:172.21.15.136:6789/0 conn(0x7f9d2806d9d0 legacy=0x7f9d281346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 >> v1:172.21.15.64:6789/0 conn(0x7f9d2813e820 legacy=0x7f9d280b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 monclient: found mon.c 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.571+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28137da0 con 0x7f9d280af1d0 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 10 monclient: _renew_subs 2024-03-29T11:25:38.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d28137ff0 con 0x7f9d280af1d0 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 10 monclient: _renew_subs 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d281d6a30 con 0x7f9d280af1d0 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d2e447640 1 librados: init done 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d1c0047e0 con 0x7f9d280af1d0 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9d1c004b70 con 0x7f9d280af1d0 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d24ff9640 10 monclient: dump: 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:38.566 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.572+0000 7f9d267fc640 4 set_mon_vals no callback set 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.573+0000 7f9d2e447640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.573+0000 7f9d2e447640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.573+0000 7f9d2e447640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.573+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d08005080 con 0x7f9d280af1d0 2024-03-29T11:25:38.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.574+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f9d1c004cd0 con 0x7f9d280af1d0 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d27fff640 10 monclient: get_auth_request con 0x7f9d1406de70 auth_method 0 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f9d1c084970 con 0x7f9d280af1d0 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d24ff9640 10 client.4301.objecter ms_dispatch 0x7f9d28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d24ff9640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d24ff9640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d24ff9640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:25:38.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.575+0000 7f9d24ff9640 10 client.4301.objecter ms_handle_connect 0x7f9d1406de70 2024-03-29T11:25:38.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.585+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d1c050250 con 0x7f9d280af1d0 2024-03-29T11:25:38.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.585+0000 7f9d24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.585+0000 7f9d24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:38.897 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:38.897 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:38.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.911+0000 7f9d2e447640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.911+0000 7f9d2e447640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.911+0000 7f9d2e447640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:25:38.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.911+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d08005330 con 0x7f9d280af1d0 2024-03-29T11:25:38.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.912+0000 7f9d24ff9640 1 -- 172.21.15.64:0/3284937956 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f9d1c052180 con 0x7f9d280af1d0 2024-03-29T11:25:38.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.912+0000 7f9d24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.912+0000 7f9d24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:38.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 10 librados: watch_flush enter 2024-03-29T11:25:38.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 10 librados: watch_flush exit 2024-03-29T11:25:38.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:25:38.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:25:38.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 >> v1:172.21.15.136:6816/2312274743 conn(0x7f9d1406de70 legacy=0x7f9d14070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 10 monclient: shutdown 2024-03-29T11:25:38.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.916+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 >> v1:172.21.15.64:6790/0 conn(0x7f9d280af1d0 legacy=0x7f9d280b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.917+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 shutdown_connections 2024-03-29T11:25:38.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.917+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 >> 172.21.15.64:0/3284937956 conn(0x7f9d2806b660 msgr2=0x7f9d2813d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.918+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 shutdown_connections 2024-03-29T11:25:38.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.918+0000 7f9d2e447640 1 -- 172.21.15.64:0/3284937956 wait complete. 2024-03-29T11:25:38.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:38.918+0000 7f9d2e447640 1 librados: shutdown 2024-03-29T11:25:38.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:45.728 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:45.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.889+0000 7fa151286640 10 monclient: get_monmap_and_config 2024-03-29T11:25:45.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.889+0000 7fa151286640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.889+0000 7fa151286640 10 monclient: monmap: 2024-03-29T11:25:45.874 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:45.874 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:45.874 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:45.890683+0000 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:45.890683+0000 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:45.875 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.893+0000 7fa151286640 1 Processor -- start 2024-03-29T11:25:45.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.893+0000 7fa151286640 1 -- start start 2024-03-29T11:25:45.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.893+0000 7fa151286640 10 monclient: init 2024-03-29T11:25:45.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.894+0000 7fa151286640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.894+0000 7fa151286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:45.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.894+0000 7fa151286640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c1457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:45.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.897+0000 7fa151286640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:45.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.898+0000 7fa151286640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.898+0000 7fa151286640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0aaef0 con 0x7fa14c13f650 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.899+0000 7fa151286640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c145bd0 con 0x7fa14c1457f0 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.899+0000 7fa151286640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c145eb0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.899+0000 7fa151286640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.899+0000 7fa151286640 10 monclient(hunting): authenticate will time out at 1282.169103s 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.899+0000 7fa14affd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa14c1457f0 0x7fa14c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:47093/0 (socket says 172.21.15.64:47093) 2024-03-29T11:25:45.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.899+0000 7fa14affd640 1 -- 172.21.15.64:0/2350493345 learned_addr learned my addr 172.21.15.64:0/2350493345 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940277912 0 0) 0x7fa14c145bd0 con 0x7fa14c1457f0 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa134003520 con 0x7fa14c1457f0 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290747321 0 0) 0x7fa14c145eb0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c145bd0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.901+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905896929 0 0) 0x7fa134003520 con 0x7fa14c1457f0 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 >> v1:172.21.15.64:6790/0 conn(0x7fa14c0a9d70 legacy=0x7fa14c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 >> v1:172.21.15.64:6789/0 conn(0x7fa14c13f650 legacy=0x7fa14c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 monclient: found mon.noname-c 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c146340 con 0x7fa14c1457f0 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa151286640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa151286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa13c001690 con 0x7fa14c1457f0 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: dump: 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:45.891 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa151286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa13c0017f0 con 0x7fa14c1457f0 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa151286640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa151286640 4 set_mon_vals no callback set 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 1 -- 172.21.15.64:0/2350493345 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa13c001c40 con 0x7fa14c1457f0 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.902+0000 7fa149ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.903+0000 7fa149ffb640 10 monclient: dump: 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:45.892 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.903+0000 7fa151286640 10 monclient: shutdown 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.903+0000 7fa151286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.903+0000 7fa151286640 1 -- 172.21.15.64:0/2350493345 >> v1:172.21.15.136:6789/0 conn(0x7fa14c1457f0 legacy=0x7fa14c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.903+0000 7fa151286640 1 -- 172.21.15.64:0/2350493345 shutdown_connections 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.903+0000 7fa151286640 1 -- 172.21.15.64:0/2350493345 >> 172.21.15.64:0/2350493345 conn(0x7fa14c066a00 msgr2=0x7fa14c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.904+0000 7fa151286640 1 -- 172.21.15.64:0/2350493345 shutdown_connections 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.907+0000 7fa151286640 1 -- 172.21.15.64:0/2350493345 wait complete. 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.908+0000 7fa151286640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.908+0000 7fa151286640 10 monclient: monmap: 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:45.895 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.908+0000 7fa151286640 1 librados: starting msgr at 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.908+0000 7fa151286640 1 librados: starting objecter 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.908+0000 7fa151286640 1 Processor -- start 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 1 -- start start 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 1 librados: setting wanted keys 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 1 librados: calling monclient init 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 10 monclient: init 2024-03-29T11:25:45.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.909+0000 7fa151286640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c1457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0ad310 con 0x7fa14c0a9d70 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c1d6f30 con 0x7fa14c1457f0 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c13fda0 con 0x7fa14c13f650 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.910+0000 7fa151286640 10 monclient(hunting): authenticate will time out at 1282.179941s 2024-03-29T11:25:45.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.914+0000 7fa14affd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa14c0a9d70 0x7fa14c1d7450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33033/0 (socket says 172.21.15.64:33033) 2024-03-29T11:25:45.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.914+0000 7fa14affd640 1 -- 172.21.15.64:0/4196930053 learned_addr learned my addr 172.21.15.64:0/4196930053 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 client.?.objecter ms_handle_connect 0x7fa14c0a9d70 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 client.?.objecter ms_handle_connect 0x7fa14c1457f0 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901625932 0 0) 0x7fa14c0ad310 con 0x7fa14c0a9d70 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.915+0000 7fa1337fe640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa138004eb0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974944391 0 0) 0x7fa14c1d6f30 con 0x7fa14c1457f0 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c0ad310 con 0x7fa14c1457f0 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1213477604 0 0) 0x7fa138004eb0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c1d6f30 con 0x7fa14c0a9d70 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 514212226 0 0) 0x7fa14c0ad310 con 0x7fa14c1457f0 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa138004eb0 con 0x7fa14c1457f0 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa13c001670 con 0x7fa14c0a9d70 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): dump: 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.903 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa140001760 con 0x7fa14c1457f0 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.916+0000 7fa1337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient(hunting): dump: 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:45.904 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1240726989 0 0) 0x7fa14c1d6f30 con 0x7fa14c0a9d70 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 >> v1:172.21.15.64:6790/0 conn(0x7fa14c13f650 legacy=0x7fa14c137ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 >> v1:172.21.15.136:6789/0 conn(0x7fa14c1457f0 legacy=0x7fa14c1e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 monclient: found mon.a 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c13ff30 con 0x7fa14c0a9d70 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa151286640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa151286640 10 monclient: _renew_subs 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa151286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa13c004a90 con 0x7fa14c0a9d70 2024-03-29T11:25:45.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c1d7b60 con 0x7fa14c0a9d70 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa13c004e60 con 0x7fa14c0a9d70 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa1337fe640 10 monclient: dump: 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:45.907 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa148ff9640 4 set_mon_vals no callback set 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.917+0000 7fa151286640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.918+0000 7fa151286640 10 monclient: _renew_subs 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.918+0000 7fa151286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.918+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c0ac1c0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.918+0000 7fa151286640 1 librados: init done 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.919+0000 7fa151286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.919+0000 7fa151286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.919+0000 7fa151286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.919+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c075f10 con 0x7fa14c0a9d70 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.925+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fa13c01ae30 con 0x7fa14c0a9d70 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.925+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fa13c084b00 con 0x7fa14c0a9d70 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.925+0000 7fa1337fe640 10 client.4290.objecter ms_dispatch 0x7fa14c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.925+0000 7fa1337fe640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.925+0000 7fa1337fe640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.926+0000 7fa1337fe640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.926+0000 7fa14a7fc640 10 monclient: get_auth_request con 0x7fa13806e110 auth_method 0 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.926+0000 7fa1337fe640 10 client.4290.objecter ms_handle_connect 0x7fa13806e110 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.930+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa13c0503e0 con 0x7fa14c0a9d70 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.930+0000 7fa1337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:45.930+0000 7fa1337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:46.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.263+0000 7fa151286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.263+0000 7fa151286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.263+0000 7fa151286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:25:46.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.263+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa14c1d70c0 con 0x7fa14c0a9d70 2024-03-29T11:25:46.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.265+0000 7fa1337fe640 1 -- 172.21.15.64:0/4196930053 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fa13c052310 con 0x7fa14c0a9d70 2024-03-29T11:25:46.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.265+0000 7fa1337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.265+0000 7fa1337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:46.251 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:46.251 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:46.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.269+0000 7fa151286640 10 librados: watch_flush enter 2024-03-29T11:25:46.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 10 librados: watch_flush exit 2024-03-29T11:25:46.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 >> v1:172.21.15.136:6816/2312274743 conn(0x7fa13806e110 legacy=0x7fa1380705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 10 monclient: shutdown 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.271+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 >> v1:172.21.15.64:6789/0 conn(0x7fa14c0a9d70 legacy=0x7fa14c1d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.272+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 shutdown_connections 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.272+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 >> 172.21.15.64:0/4196930053 conn(0x7fa14c066a00 msgr2=0x7fa14c0b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.272+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 shutdown_connections 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.272+0000 7fa151286640 1 -- 172.21.15.64:0/4196930053 wait complete. 2024-03-29T11:25:46.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:46.272+0000 7fa151286640 1 librados: shutdown 2024-03-29T11:25:46.275 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:53.081 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:53.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.248+0000 7fcb745c8640 10 monclient: get_monmap_and_config 2024-03-29T11:25:53.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.248+0000 7fcb745c8640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.248+0000 7fcb745c8640 10 monclient: monmap: 2024-03-29T11:25:53.234 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:25:53.249613+0000 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:25:53.249613+0000 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:25:53.235 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:53.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.250+0000 7fcb745c8640 1 Processor -- start 2024-03-29T11:25:53.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.250+0000 7fcb745c8640 1 -- start start 2024-03-29T11:25:53.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.250+0000 7fcb745c8640 10 monclient: init 2024-03-29T11:25:53.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:53.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c0a9930 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:53.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c149dc0 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:53.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6c14ff40 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:53.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0aae60 con 0x7fcb6c14ff40 2024-03-29T11:25:53.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0b4310 con 0x7fcb6c0a9930 2024-03-29T11:25:53.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0afdd0 con 0x7fcb6c149dc0 2024-03-29T11:25:53.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.252+0000 7fcb745c8640 10 monclient(hunting): authenticate will time out at 1289.522079s 2024-03-29T11:25:53.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.256+0000 7fcb72b65640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcb6c14ff40 0x7fcb6c152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:44555/0 (socket says 172.21.15.64:44555) 2024-03-29T11:25:53.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.256+0000 7fcb72b65640 1 -- 172.21.15.64:0/73542710 learned_addr learned my addr 172.21.15.64:0/73542710 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:53.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385157599 0 0) 0x7fcb6c0aae60 con 0x7fcb6c14ff40 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5c003520 con 0x7fcb6c14ff40 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2773115978 0 0) 0x7fcb5c003520 con 0x7fcb6c14ff40 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 >> v1:172.21.15.64:6790/0 conn(0x7fcb6c149dc0 legacy=0x7fcb6c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:53.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.257+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 >> v1:172.21.15.136:6789/0 conn(0x7fcb6c0a9930 legacy=0x7fcb6c143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 1 monclient: found mon.noname-a 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c14a950 con 0x7fcb6c14ff40 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcb58001a10 con 0x7fcb6c14ff40 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.260+0000 7fcb71362640 10 monclient: dump: 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb58001c70 con 0x7fcb6c14ff40 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb71362640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb71362640 1 -- 172.21.15.64:0/73542710 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcb580020c0 con 0x7fcb6c14ff40 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb71362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb71362640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb71362640 10 monclient: dump: 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:53.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb745c8640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:25:53.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb745c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:53.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.261+0000 7fcb745c8640 4 set_mon_vals no callback set 2024-03-29T11:25:53.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.262+0000 7fcb745c8640 10 monclient: shutdown 2024-03-29T11:25:53.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.262+0000 7fcb745c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.262+0000 7fcb745c8640 1 -- 172.21.15.64:0/73542710 >> v1:172.21.15.64:6789/0 conn(0x7fcb6c14ff40 legacy=0x7fcb6c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.264+0000 7fcb745c8640 1 -- 172.21.15.64:0/73542710 shutdown_connections 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.264+0000 7fcb745c8640 1 -- 172.21.15.64:0/73542710 >> 172.21.15.64:0/73542710 conn(0x7fcb6c0b35e0 msgr2=0x7fcb6c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.264+0000 7fcb745c8640 1 -- 172.21.15.64:0/73542710 shutdown_connections 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.265+0000 7fcb745c8640 1 -- 172.21.15.64:0/73542710 wait complete. 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.266+0000 7fcb745c8640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.266+0000 7fcb745c8640 10 monclient: monmap: 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:53.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 1 librados: starting msgr at 2024-03-29T11:25:53.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 1 librados: starting objecter 2024-03-29T11:25:53.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 1 Processor -- start 2024-03-29T11:25:53.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 1 -- start start 2024-03-29T11:25:53.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 1 librados: setting wanted keys 2024-03-29T11:25:53.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 1 librados: calling monclient init 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.267+0000 7fcb745c8640 10 monclient: init 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.268+0000 7fcb745c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.268+0000 7fcb745c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c0a9930 addr v1:172.21.15.136:6789/0 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6c149dc0 addr v1:172.21.15.64:6789/0 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c14ff40 addr v1:172.21.15.64:6790/0 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c1478d0 con 0x7fcb6c149dc0 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c1db840 con 0x7fcb6c0a9930 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c1db4f0 con 0x7fcb6c14ff40 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.271+0000 7fcb745c8640 10 monclient(hunting): authenticate will time out at 1289.541861s 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.272+0000 7fcb72364640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fcb6c0a9930 0x7fcb6c1dbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:38787/0 (socket says 172.21.15.64:38787) 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.272+0000 7fcb72364640 1 -- 172.21.15.64:0/2104025004 learned_addr learned my addr 172.21.15.64:0/2104025004 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:25:53.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 10 client.?.objecter ms_handle_connect 0x7fcb6c0a9930 2024-03-29T11:25:53.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342195549 0 0) 0x7fcb6c1db840 con 0x7fcb6c0a9930 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.275+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb50004eb0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188061696 0 0) 0x7fcb50004eb0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb6c1db840 con 0x7fcb6c0a9930 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcb64001870 con 0x7fcb6c0a9930 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient(hunting): dump: 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570583491 0 0) 0x7fcb6c1db840 con 0x7fcb6c0a9930 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 >> v1:172.21.15.64:6790/0 conn(0x7fcb6c14ff40 legacy=0x7fcb6c1e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 >> v1:172.21.15.64:6789/0 conn(0x7fcb6c149dc0 legacy=0x7fcb6c13a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 monclient: found mon.b 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c149010 con 0x7fcb6c0a9930 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.276+0000 7fcb6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb640019d0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcb64004ae0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb6affd640 10 monclient: dump: 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:25:53.267 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb745c8640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb745c8640 10 monclient: _renew_subs 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb745c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.277+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c147ec0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.278+0000 7fcb745c8640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.278+0000 7fcb745c8640 10 monclient: _renew_subs 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.278+0000 7fcb745c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.278+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c13acd0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.278+0000 7fcb745c8640 1 librados: init done 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.278+0000 7fcb6bfff640 4 set_mon_vals no callback set 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.280+0000 7fcb745c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.280+0000 7fcb745c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb745c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fcb6401aab0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb54005080 con 0x7fcb6c0a9930 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fcb64084b10 con 0x7fcb6c0a9930 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb6affd640 10 client.4306.objecter ms_dispatch 0x7fcb6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb6affd640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb6affd640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.281+0000 7fcb6affd640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.282+0000 7fcb71b63640 10 monclient: get_auth_request con 0x7fcb5006dcd0 auth_method 0 2024-03-29T11:25:53.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.282+0000 7fcb6affd640 10 client.4306.objecter ms_handle_connect 0x7fcb5006dcd0 2024-03-29T11:25:53.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.291+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb640503f0 con 0x7fcb6c0a9930 2024-03-29T11:25:53.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.291+0000 7fcb6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.291+0000 7fcb6affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:53.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.619+0000 7fcb745c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.619+0000 7fcb745c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.619+0000 7fcb745c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:25:53.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.619+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb54005330 con 0x7fcb6c0a9930 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.621+0000 7fcb6affd640 1 -- 172.21.15.64:0/2104025004 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fcb64052320 con 0x7fcb6c0a9930 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.621+0000 7fcb6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.621+0000 7fcb6affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:53.607 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:25:53.608 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.625+0000 7fcb745c8640 10 librados: watch_flush enter 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.625+0000 7fcb745c8640 10 librados: watch_flush exit 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.625+0000 7fcb745c8640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.625+0000 7fcb745c8640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 >> v1:172.21.15.136:6816/2312274743 conn(0x7fcb5006dcd0 legacy=0x7fcb50070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 10 monclient: shutdown 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 >> v1:172.21.15.136:6789/0 conn(0x7fcb6c0a9930 legacy=0x7fcb6c1dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 shutdown_connections 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 >> 172.21.15.64:0/2104025004 conn(0x7fcb6c0b35e0 msgr2=0x7fcb6c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 shutdown_connections 2024-03-29T11:25:53.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 -- 172.21.15.64:0/2104025004 wait complete. 2024-03-29T11:25:53.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:25:53.626+0000 7fcb745c8640 1 librados: shutdown 2024-03-29T11:25:53.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:00.438 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:00.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.600+0000 7f8b25efb640 10 monclient: get_monmap_and_config 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.600+0000 7f8b25efb640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.600+0000 7f8b25efb640 10 monclient: monmap: 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:00.602509+0000 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:00.602509+0000 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:00.588 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.602+0000 7f8b25efb640 1 Processor -- start 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.602+0000 7f8b25efb640 1 -- start start 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.602+0000 7f8b25efb640 10 monclient: init 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient(hunting): picked mon.noname-c con 0x7f8b200a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:00.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient(hunting): picked mon.noname-a con 0x7f8b2013ee90 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient(hunting): picked mon.noname-b con 0x7f8b200af900 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200afd10 con 0x7f8b2013ee90 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200af6f0 con 0x7f8b200a9ac0 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200aac80 con 0x7f8b200af900 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.603+0000 7f8b25efb640 10 monclient(hunting): authenticate will time out at 1296.874480s 2024-03-29T11:26:00.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.608+0000 7f8b1effd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8b2013ee90 0x7f8b2013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45915/0 (socket says 172.21.15.64:45915) 2024-03-29T11:26:00.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.608+0000 7f8b1effd640 1 -- 172.21.15.64:0/2087959096 learned_addr learned my addr 172.21.15.64:0/2087959096 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:00.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832846063 0 0) 0x7f8b200aac80 con 0x7f8b200af900 2024-03-29T11:26:00.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b08003520 con 0x7f8b200af900 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658005837 0 0) 0x7f8b200afd10 con 0x7f8b2013ee90 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.610+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b200aac80 con 0x7f8b2013ee90 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1443557543 0 0) 0x7f8b200aac80 con 0x7f8b2013ee90 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 >> v1:172.21.15.64:6790/0 conn(0x7f8b200af900 legacy=0x7f8b20149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 >> v1:172.21.15.136:6789/0 conn(0x7f8b200a9ac0 legacy=0x7f8b200adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b200aaf60 con 0x7f8b2013ee90 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8b10001670 con 0x7f8b2013ee90 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: dump: 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b25efb640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b25efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b100018c0 con 0x7f8b2013ee90 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 1 -- 172.21.15.64:0/2087959096 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8b10001c50 con 0x7f8b2013ee90 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b25efb640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b25efb640 4 set_mon_vals no callback set 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.611+0000 7f8b1e7fc640 10 monclient: dump: 2024-03-29T11:26:00.602 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 10 monclient: shutdown 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 1 -- 172.21.15.64:0/2087959096 >> v1:172.21.15.64:6789/0 conn(0x7f8b2013ee90 legacy=0x7f8b2013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 1 -- 172.21.15.64:0/2087959096 shutdown_connections 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 1 -- 172.21.15.64:0/2087959096 >> 172.21.15.64:0/2087959096 conn(0x7f8b200b35e0 msgr2=0x7f8b200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 1 -- 172.21.15.64:0/2087959096 shutdown_connections 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.612+0000 7f8b25efb640 1 -- 172.21.15.64:0/2087959096 wait complete. 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.613+0000 7f8b25efb640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.613+0000 7f8b25efb640 10 monclient: monmap: 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.614+0000 7f8b25efb640 1 librados: starting msgr at 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.614+0000 7f8b25efb640 1 librados: starting objecter 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.614+0000 7f8b25efb640 1 Processor -- start 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.615+0000 7f8b25efb640 1 -- start start 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.615+0000 7f8b25efb640 1 librados: setting wanted keys 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.615+0000 7f8b25efb640 1 librados: calling monclient init 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.615+0000 7f8b25efb640 10 monclient: init 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient(hunting): picked mon.noname-b con 0x7f8b200a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:00.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient(hunting): picked mon.noname-c con 0x7f8b200af900 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient(hunting): picked mon.noname-a con 0x7f8b2013ee90 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20070ec0 con 0x7f8b2013ee90 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2006f610 con 0x7f8b200a9ac0 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20070240 con 0x7f8b200af900 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.616+0000 7f8b25efb640 10 monclient(hunting): authenticate will time out at 1296.887155s 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.620+0000 7f8b1ffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8b2013ee90 0x7f8b201e6240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:46047/0 (socket says 172.21.15.64:46047) 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.620+0000 7f8b1ffff640 1 -- 172.21.15.64:0/373432511 learned_addr learned my addr 172.21.15.64:0/373432511 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 client.?.objecter ms_handle_connect 0x7f8b2013ee90 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189234127 0 0) 0x7f8b20070ec0 con 0x7f8b2013ee90 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b0c004eb0 con 0x7f8b2013ee90 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018448310 0 0) 0x7f8b0c004eb0 con 0x7f8b2013ee90 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b20070ec0 con 0x7f8b2013ee90 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8b18001c30 con 0x7f8b2013ee90 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): dump: 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1158268398 0 0) 0x7f8b20070ec0 con 0x7f8b2013ee90 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 >> v1:172.21.15.64:6790/0 conn(0x7f8b200af900 legacy=0x7f8b2006fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 >> v1:172.21.15.136:6789/0 conn(0x7f8b200a9ac0 legacy=0x7f8b2006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 monclient: found mon.a 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20071130 con 0x7f8b2013ee90 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.621+0000 7f8b07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b18003990 con 0x7f8b2013ee90 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8b18003de0 con 0x7f8b2013ee90 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b07fff640 10 monclient: dump: 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:00.612 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:00.613 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:00.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 10 monclient: _renew_subs 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b20123c40 con 0x7f8b2013ee90 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 10 monclient: _renew_subs 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b20071830 con 0x7f8b2013ee90 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.622+0000 7f8b25efb640 1 librados: init done 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.623+0000 7f8b1dffb640 4 set_mon_vals no callback set 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.624+0000 7f8b25efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.624+0000 7f8b25efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.624+0000 7f8b25efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:00.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.624+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b00005080 con 0x7f8b2013ee90 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.629+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f8b18003f40 con 0x7f8b2013ee90 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.630+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f8b18054b20 con 0x7f8b2013ee90 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.630+0000 7f8b07fff640 10 client.4299.objecter ms_dispatch 0x7f8b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.630+0000 7f8b07fff640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.630+0000 7f8b07fff640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.630+0000 7f8b07fff640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.631+0000 7f8b1f7fe640 10 monclient: get_auth_request con 0x7f8b0c06dc50 auth_method 0 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.632+0000 7f8b07fff640 10 client.4299.objecter ms_handle_connect 0x7f8b0c06dc50 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.634+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b18050dc0 con 0x7f8b2013ee90 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.634+0000 7f8b07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.634+0000 7f8b07fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:00.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.957+0000 7f8b25efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.957+0000 7f8b25efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.957+0000 7f8b25efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:00.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.957+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b00005330 con 0x7f8b2013ee90 2024-03-29T11:26:00.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.959+0000 7f8b07fff640 1 -- 172.21.15.64:0/373432511 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f8b18050dc0 con 0x7f8b2013ee90 2024-03-29T11:26:00.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.959+0000 7f8b07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.959+0000 7f8b07fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:00.946 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:00.946 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 10 librados: watch_flush enter 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 10 librados: watch_flush exit 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 >> v1:172.21.15.136:6816/2312274743 conn(0x7f8b0c06dc50 legacy=0x7f8b0c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 10 monclient: shutdown 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.963+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 >> v1:172.21.15.64:6789/0 conn(0x7f8b2013ee90 legacy=0x7f8b201e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.964+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 shutdown_connections 2024-03-29T11:26:00.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.964+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 >> 172.21.15.64:0/373432511 conn(0x7f8b200b35e0 msgr2=0x7f8b20131fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.964+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 shutdown_connections 2024-03-29T11:26:00.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.964+0000 7f8b25efb640 1 -- 172.21.15.64:0/373432511 wait complete. 2024-03-29T11:26:00.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:00.964+0000 7f8b25efb640 1 librados: shutdown 2024-03-29T11:26:00.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:07.773 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.940+0000 7f9601e7b640 10 monclient: get_monmap_and_config 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.940+0000 7f9601e7b640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.940+0000 7f9601e7b640 10 monclient: monmap: 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:07.941327+0000 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:07.941327+0000 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.926 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:07.927 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:07.927 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:07.927 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:07.927 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:07.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.942+0000 7f9601e7b640 1 Processor -- start 2024-03-29T11:26:07.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.942+0000 7f9601e7b640 1 -- start start 2024-03-29T11:26:07.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.942+0000 7f9601e7b640 10 monclient: init 2024-03-29T11:26:07.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:07.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f95fc071de0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:07.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f95fc0721c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:07.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f95fc145ae0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:07.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc145650 con 0x7f95fc071de0 2024-03-29T11:26:07.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc0ae260 con 0x7f95fc0721c0 2024-03-29T11:26:07.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc1420d0 con 0x7f95fc145ae0 2024-03-29T11:26:07.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.943+0000 7f9601e7b640 10 monclient(hunting): authenticate will time out at 1304.213327s 2024-03-29T11:26:07.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.947+0000 7f95fb7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f95fc071de0 0x7f95fc0aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45695/0 (socket says 172.21.15.64:45695) 2024-03-29T11:26:07.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.947+0000 7f95fb7fe640 1 -- 172.21.15.64:0/3409534707 learned_addr learned my addr 172.21.15.64:0/3409534707 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:07.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.951+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114666943 0 0) 0x7f95fc145650 con 0x7f95fc071de0 2024-03-29T11:26:07.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e4003520 con 0x7f95fc071de0 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508633136 0 0) 0x7f95fc0ae260 con 0x7f95fc0721c0 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.952+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc145650 con 0x7f95fc0721c0 2024-03-29T11:26:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27033579 0 0) 0x7f95fc1420d0 con 0x7f95fc145ae0 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc0ae260 con 0x7f95fc145ae0 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1712383584 0 0) 0x7f95fc145650 con 0x7f95fc0721c0 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 >> v1:172.21.15.64:6790/0 conn(0x7f95fc145ae0 legacy=0x7f95fc145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 >> v1:172.21.15.64:6789/0 conn(0x7f95fc071de0 legacy=0x7f95fc0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 monclient: found mon.noname-c 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc0b0590 con 0x7f95fc0721c0 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f95ec0015f0 con 0x7f95fc0721c0 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:07.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f95fa7fc640 10 monclient: dump: 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f9601e7b640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.953+0000 7f9601e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f95fa7fc640 1 -- 172.21.15.64:0/3409534707 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95ec0017d0 con 0x7f95fc0721c0 2024-03-29T11:26:07.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f95fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f9601e7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f9601e7b640 4 set_mon_vals no callback set 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f9601e7b640 10 monclient: shutdown 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f9601e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.954+0000 7f9601e7b640 1 -- 172.21.15.64:0/3409534707 >> v1:172.21.15.136:6789/0 conn(0x7f95fc0721c0 legacy=0x7f95fc13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.955+0000 7f9601e7b640 1 -- 172.21.15.64:0/3409534707 shutdown_connections 2024-03-29T11:26:07.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.955+0000 7f9601e7b640 1 -- 172.21.15.64:0/3409534707 >> 172.21.15.64:0/3409534707 conn(0x7f95fc06b660 msgr2=0x7f95fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.955+0000 7f9601e7b640 1 -- 172.21.15.64:0/3409534707 shutdown_connections 2024-03-29T11:26:07.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.955+0000 7f9601e7b640 1 -- 172.21.15.64:0/3409534707 wait complete. 2024-03-29T11:26:07.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 10 monclient: monmap: 2024-03-29T11:26:07.946 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:07.946 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 1 librados: starting msgr at 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 1 librados: starting objecter 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 1 Processor -- start 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 1 -- start start 2024-03-29T11:26:07.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 1 librados: setting wanted keys 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 1 librados: calling monclient init 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.957+0000 7f9601e7b640 10 monclient: init 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f95fc071de0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f95fc145ae0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f95fc1357c0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc1331e0 con 0x7f95fc071de0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc1340b0 con 0x7f95fc145ae0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc134390 con 0x7f95fc1357c0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.958+0000 7f9601e7b640 10 monclient(hunting): authenticate will time out at 1304.228512s 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.963+0000 7f95faffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f95fc145ae0 0x7f95fc132ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:35261/0 (socket says 172.21.15.64:35261) 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.963+0000 7f95faffd640 1 -- 172.21.15.64:0/3807575781 learned_addr learned my addr 172.21.15.64:0/3807575781 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.963+0000 7f95e3fff640 10 client.?.objecter ms_handle_connect 0x7f95fc145ae0 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.963+0000 7f95e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975632584 0 0) 0x7f95fc1340b0 con 0x7f95fc145ae0 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e8004eb0 con 0x7f95fc145ae0 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 971188458 0 0) 0x7f95e8004eb0 con 0x7f95fc145ae0 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e8005300 con 0x7f95fc145ae0 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f95ec003810 con 0x7f95fc145ae0 2024-03-29T11:26:07.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): dump: 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:07.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.964+0000 7f95e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 129835352 0 0) 0x7f95e8005300 con 0x7f95fc145ae0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 >> v1:172.21.15.64:6790/0 conn(0x7f95fc1357c0 legacy=0x7f95fc12de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 >> v1:172.21.15.64:6789/0 conn(0x7f95fc071de0 legacy=0x7f95fc131f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 1 monclient: found mon.b 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.965+0000 7f95e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc08bd40 con 0x7f95fc145ae0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 10 client.?.objecter ms_handle_connect 0x7f95fc1357c0 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:26:07.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 10 monclient: _renew_subs 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc1374c0 con 0x7f95fc145ae0 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 10 monclient: _renew_subs 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc12f5a0 con 0x7f95fc145ae0 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f9601e7b640 1 librados: init done 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95ec0019c0 con 0x7f95fc145ae0 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.966+0000 7f95f97fa640 4 set_mon_vals no callback set 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.967+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f95ec003810 con 0x7f95fc145ae0 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.967+0000 7f95e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.967+0000 7f95e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:07.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.967+0000 7f95e3fff640 10 monclient: dump: 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.969+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f95ec003dc0 con 0x7f95fc145ae0 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f95ec084890 con 0x7f95fc145ae0 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f95e3fff640 10 client.4312.objecter ms_dispatch 0x7f95fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f95e3fff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:07.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f95e3fff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f95e3fff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f9601e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f9601e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f9601e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.970+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95dc005080 con 0x7f95fc145ae0 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.971+0000 7f95fb7fe640 10 monclient: get_auth_request con 0x7f95e8076a50 auth_method 0 2024-03-29T11:26:07.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.974+0000 7f95e3fff640 10 client.4312.objecter ms_handle_connect 0x7f95e8076a50 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.981+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95ec0501d0 con 0x7f95fc145ae0 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.981+0000 7f95e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:07.981+0000 7f95e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:08.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.324+0000 7f9601e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.324+0000 7f9601e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.324+0000 7f9601e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:08.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.324+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95dc005330 con 0x7f95fc145ae0 2024-03-29T11:26:08.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.325+0000 7f95e3fff640 1 -- 172.21.15.64:0/3807575781 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f95ec003670 con 0x7f95fc145ae0 2024-03-29T11:26:08.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.325+0000 7f95e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.325+0000 7f95e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:08.311 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:08.311 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:08.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.329+0000 7f9601e7b640 10 librados: watch_flush enter 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.329+0000 7f9601e7b640 10 librados: watch_flush exit 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.329+0000 7f9601e7b640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.329+0000 7f9601e7b640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 >> v1:172.21.15.136:6816/2312274743 conn(0x7f95e8076a50 legacy=0x7f95e8078f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 10 monclient: shutdown 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 >> v1:172.21.15.136:6789/0 conn(0x7f95fc145ae0 legacy=0x7f95fc132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 shutdown_connections 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 >> 172.21.15.64:0/3807575781 conn(0x7f95fc06b660 msgr2=0x7f95fc141900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 shutdown_connections 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 -- 172.21.15.64:0/3807575781 wait complete. 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:08.330+0000 7f9601e7b640 1 librados: shutdown 2024-03-29T11:26:08.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:15.139 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.300+0000 7f1817eae640 10 monclient: get_monmap_and_config 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.300+0000 7f1817eae640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.300+0000 7f1817eae640 10 monclient: monmap: 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:15.301010+0000 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:15.301010+0000 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.286 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:15.287 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:15.287 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:15.287 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.301+0000 7f1817eae640 1 Processor -- start 2024-03-29T11:26:15.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.302+0000 7f1817eae640 1 -- start start 2024-03-29T11:26:15.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.302+0000 7f1817eae640 10 monclient: init 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient(hunting): picked mon.noname-b con 0x7f1808006a70 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient(hunting): picked mon.noname-a con 0x7f180802b100 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient(hunting): picked mon.noname-c con 0x7f1808029c30 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180802a270 con 0x7f180802b100 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180802c440 con 0x7f1808029c30 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080c9440 con 0x7f1808006a70 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.303+0000 7f1817eae640 10 monclient(hunting): authenticate will time out at 1311.573404s 2024-03-29T11:26:15.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.310+0000 7f1815449640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f180802b100 0x7f18080cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40891/0 (socket says 172.21.15.64:40891) 2024-03-29T11:26:15.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.310+0000 7f1815449640 1 -- 172.21.15.64:0/2553153508 learned_addr learned my addr 172.21.15.64:0/2553153508 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:15.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647904307 0 0) 0x7f18080c9440 con 0x7f1808006a70 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17fc003520 con 0x7f1808006a70 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611770290 0 0) 0x7f180802a270 con 0x7f180802b100 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.311+0000 7f1814c48640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18080c9440 con 0x7f180802b100 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674708580 0 0) 0x7f180802c440 con 0x7f1808029c30 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f180802a270 con 0x7f1808029c30 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 182463993 0 0) 0x7f17fc003520 con 0x7f1808006a70 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 >> v1:172.21.15.136:6789/0 conn(0x7f1808029c30 legacy=0x7f18080d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 >> v1:172.21.15.64:6789/0 conn(0x7f180802b100 legacy=0x7f18080cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 monclient: found mon.noname-b 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18080c95d0 con 0x7f1808006a70 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.312+0000 7f1814c48640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1814c48640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1814c48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1817eae640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1817eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f180c001610 con 0x7f1808006a70 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1814c48640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1814c48640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1814c48640 10 monclient: dump: 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:15.302 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.313+0000 7f1817eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f180c001830 con 0x7f1808006a70 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1814c48640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1814c48640 1 -- 172.21.15.64:0/2553153508 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f180c001bc0 con 0x7f1808006a70 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1814c48640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1814c48640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1814c48640 10 monclient: dump: 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1817eae640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.314+0000 7f1817eae640 4 set_mon_vals no callback set 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.315+0000 7f1817eae640 10 monclient: shutdown 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.315+0000 7f1817eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.315+0000 7f1817eae640 1 -- 172.21.15.64:0/2553153508 >> v1:172.21.15.64:6790/0 conn(0x7f1808006a70 legacy=0x7f18080c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.317+0000 7f1817eae640 1 -- 172.21.15.64:0/2553153508 shutdown_connections 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.317+0000 7f1817eae640 1 -- 172.21.15.64:0/2553153508 >> 172.21.15.64:0/2553153508 conn(0x7f18080058f0 msgr2=0x7f1808005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.319+0000 7f1817eae640 1 -- 172.21.15.64:0/2553153508 shutdown_connections 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.319+0000 7f1817eae640 1 -- 172.21.15.64:0/2553153508 wait complete. 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.320+0000 7f1817eae640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.320+0000 7f1817eae640 10 monclient: monmap: 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.320+0000 7f1817eae640 1 librados: starting msgr at 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.320+0000 7f1817eae640 1 librados: starting objecter 2024-03-29T11:26:15.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.321+0000 7f1817eae640 1 Processor -- start 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.322+0000 7f1817eae640 1 -- start start 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.322+0000 7f1817eae640 1 librados: setting wanted keys 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.322+0000 7f1817eae640 1 librados: calling monclient init 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.322+0000 7f1817eae640 10 monclient: init 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:15.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 10 monclient(hunting): picked mon.noname-a con 0x7f1808006a70 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 10 monclient(hunting): picked mon.noname-b con 0x7f1808029c30 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 10 monclient(hunting): picked mon.noname-c con 0x7f1808155860 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1808155d70 con 0x7f1808006a70 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.323+0000 7f1817eae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18081574d0 con 0x7f1808029c30 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.324+0000 7f1817eae640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18081577b0 con 0x7f1808155860 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.324+0000 7f1817eae640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.324+0000 7f1817eae640 10 monclient(hunting): authenticate will time out at 1311.593883s 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.329+0000 7f1815c4a640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1808006a70 0x7f1808157bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41205/0 (socket says 172.21.15.64:41205) 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.329+0000 7f1815c4a640 1 -- 172.21.15.64:0/3204233326 learned_addr learned my addr 172.21.15.64:0/3204233326 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 10 client.?.objecter ms_handle_connect 0x7f1808006a70 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679606726 0 0) 0x7f1808155d70 con 0x7f1808006a70 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f4004eb0 con 0x7f1808006a70 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.330+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597043938 0 0) 0x7f17f4004eb0 con 0x7f1808006a70 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1808155d70 con 0x7f1808006a70 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f180c001bc0 con 0x7f1808006a70 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): dump: 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 client.?.objecter ms_handle_connect 0x7f1808029c30 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535855986 0 0) 0x7f18081574d0 con 0x7f1808029c30 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.331+0000 7f18067fc640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f4004eb0 con 0x7f1808029c30 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 10 client.?.objecter ms_handle_connect 0x7f1808155860 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378532057 0 0) 0x7f18081577b0 con 0x7f1808155860 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18081574d0 con 0x7f1808155860 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109061253 0 0) 0x7f17f4004eb0 con 0x7f1808029c30 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.332+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18081577b0 con 0x7f1808029c30 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.333+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1800001930 con 0x7f1808029c30 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.333+0000 7f18067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.333+0000 7f18067fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.333+0000 7f18067fc640 10 monclient(hunting): dump: 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.324 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.324 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.324 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:15.324 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:15.324 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:15.324 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3624957068 0 0) 0x7f1808155d70 con 0x7f1808006a70 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 >> v1:172.21.15.64:6790/0 conn(0x7f1808155860 legacy=0x7f180816b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 >> v1:172.21.15.136:6789/0 conn(0x7f1808029c30 legacy=0x7f1808155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 1 monclient: found mon.a 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18081582e0 con 0x7f1808006a70 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f18067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 10 monclient: _renew_subs 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18080c22c0 con 0x7f1808006a70 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 10 monclient: _renew_subs 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18080c2660 con 0x7f1808006a70 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.334+0000 7f1817eae640 1 librados: init done 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f180c001d20 con 0x7f1808006a70 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f18067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f1817eae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f180c003a40 con 0x7f1808006a70 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f18067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f18067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f18067fc640 10 monclient: dump: 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:15.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f1817eae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f1817eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.337+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1808004cd0 con 0x7f1808006a70 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.338+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f180c003c60 con 0x7f1808006a70 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.338+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f180c084b50 con 0x7f1808006a70 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.338+0000 7f18077fe640 4 set_mon_vals no callback set 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.339+0000 7f18067fc640 10 client.4308.objecter ms_dispatch 0x7f1810000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.339+0000 7f18067fc640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.339+0000 7f18067fc640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.339+0000 7f18067fc640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.339+0000 7f1815449640 10 monclient: get_auth_request con 0x7f17f406dee0 auth_method 0 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.340+0000 7f18067fc640 10 client.4308.objecter ms_handle_connect 0x7f17f406dee0 2024-03-29T11:26:15.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.348+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f180c050430 con 0x7f1808006a70 2024-03-29T11:26:15.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.348+0000 7f18067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.348+0000 7f18067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:15.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.674+0000 7f1817eae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.674+0000 7f1817eae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.674+0000 7f1817eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:15.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.674+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1808158860 con 0x7f1808006a70 2024-03-29T11:26:15.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.675+0000 7f18067fc640 1 -- 172.21.15.64:0/3204233326 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f180c052360 con 0x7f1808006a70 2024-03-29T11:26:15.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.675+0000 7f18067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.675+0000 7f18067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:15.662 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:15.662 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:15.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 10 librados: watch_flush enter 2024-03-29T11:26:15.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 10 librados: watch_flush exit 2024-03-29T11:26:15.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:26:15.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 >> v1:172.21.15.136:6816/2312274743 conn(0x7f17f406dee0 legacy=0x7f17f40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 10 monclient: shutdown 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 >> v1:172.21.15.64:6789/0 conn(0x7f1808006a70 legacy=0x7f1808157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 shutdown_connections 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.680+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 >> 172.21.15.64:0/3204233326 conn(0x7f18080058f0 msgr2=0x7f1808037220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.681+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 shutdown_connections 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.681+0000 7f1817eae640 1 -- 172.21.15.64:0/3204233326 wait complete. 2024-03-29T11:26:15.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:15.681+0000 7f1817eae640 1 librados: shutdown 2024-03-29T11:26:15.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:22.491 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:22.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.659+0000 7f5cd0c74640 10 monclient: get_monmap_and_config 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.659+0000 7f5cd0c74640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.659+0000 7f5cd0c74640 10 monclient: monmap: 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:22.660790+0000 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:22.660790+0000 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:22.646 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.660+0000 7f5cd0c74640 1 Processor -- start 2024-03-29T11:26:22.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.661+0000 7f5cd0c74640 1 -- start start 2024-03-29T11:26:22.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.661+0000 7f5cd0c74640 10 monclient: init 2024-03-29T11:26:22.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient(hunting): picked mon.noname-b con 0x7f5ccc071c30 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient(hunting): picked mon.noname-a con 0x7f5ccc0a9750 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient(hunting): picked mon.noname-c con 0x7f5ccc13e1e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0ab300 con 0x7f5ccc0a9750 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0ac2f0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0ad8d0 con 0x7f5ccc071c30 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cd0c74640 10 monclient(hunting): authenticate will time out at 1318.932132s 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cc9d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5ccc0a9750 0x7f5ccc13dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:49722/0 (socket says 172.21.15.64:49722) 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.662+0000 7f5cc9d9b640 1 -- 172.21.15.64:0/2906752240 learned_addr learned my addr 172.21.15.64:0/2906752240 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426119509 0 0) 0x7f5ccc0ac2f0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb8003520 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804300912 0 0) 0x7f5ccc0ad8d0 con 0x7f5ccc071c30 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc0ac2f0 con 0x7f5ccc071c30 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103486587 0 0) 0x7f5ccc0ab300 con 0x7f5ccc0a9750 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc0ad8d0 con 0x7f5ccc0a9750 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2791038597 0 0) 0x7f5cb8003520 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 >> v1:172.21.15.64:6790/0 conn(0x7f5ccc071c30 legacy=0x7f5ccc0b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.663+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 >> v1:172.21.15.64:6789/0 conn(0x7f5ccc0a9750 legacy=0x7f5ccc13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 1 monclient: found mon.noname-c 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0ada60 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5cc4001a10 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: dump: 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cd0c74640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cd0c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cc4001b70 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 1 -- 172.21.15.64:0/2906752240 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5cc4001f80 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cd0c74640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cd0c74640 4 set_mon_vals no callback set 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.664+0000 7f5cc959a640 10 monclient: dump: 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 10 monclient: shutdown 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 1 -- 172.21.15.64:0/2906752240 >> v1:172.21.15.136:6789/0 conn(0x7f5ccc13e1e0 legacy=0x7f5ccc0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 1 -- 172.21.15.64:0/2906752240 shutdown_connections 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 1 -- 172.21.15.64:0/2906752240 >> 172.21.15.64:0/2906752240 conn(0x7f5ccc06b660 msgr2=0x7f5ccc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 1 -- 172.21.15.64:0/2906752240 shutdown_connections 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.665+0000 7f5cd0c74640 1 -- 172.21.15.64:0/2906752240 wait complete. 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 10 monclient: monmap: 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 1 librados: starting msgr at 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 1 librados: starting objecter 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 1 Processor -- start 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 1 -- start start 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 1 librados: setting wanted keys 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 1 librados: calling monclient init 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.667+0000 7f5cd0c74640 10 monclient: init 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.668+0000 7f5cd0c74640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.668+0000 7f5cd0c74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.668+0000 7f5cd0c74640 10 monclient(hunting): picked mon.noname-a con 0x7f5ccc071c30 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.668+0000 7f5cd0c74640 10 monclient(hunting): picked mon.noname-c con 0x7f5ccc0a9750 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cd0c74640 10 monclient(hunting): picked mon.noname-b con 0x7f5ccc13e1e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cd0c74640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc12f060 con 0x7f5ccc071c30 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cd0c74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc1e01c0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cd0c74640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc12fbf0 con 0x7f5ccc0a9750 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cd0c74640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cd0c74640 10 monclient(hunting): authenticate will time out at 1318.938945s 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cca59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5ccc071c30 0x7f5ccc126650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:49880/0 (socket says 172.21.15.64:49880) 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cca59c640 1 -- 172.21.15.64:0/3247542291 learned_addr learned my addr 172.21.15.64:0/3247542291 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 client.?.objecter ms_handle_connect 0x7f5ccc13e1e0 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 client.?.objecter ms_handle_connect 0x7f5ccc0a9750 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 client.?.objecter ms_handle_connect 0x7f5ccc071c30 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121387165 0 0) 0x7f5ccc1e01c0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.669+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cac004eb0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136181265 0 0) 0x7f5ccc12f060 con 0x7f5ccc071c30 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc1e01c0 con 0x7f5ccc071c30 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3945806552 0 0) 0x7f5cac004eb0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ccc12f060 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5cc4001e80 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983496205 0 0) 0x7f5ccc12fbf0 con 0x7f5ccc0a9750 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cac004eb0 con 0x7f5ccc0a9750 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3090371175 0 0) 0x7f5ccc1e01c0 con 0x7f5ccc071c30 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ccc12fbf0 con 0x7f5ccc071c30 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5cb40018e0 con 0x7f5ccc071c30 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 20575911 0 0) 0x7f5ccc12f060 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 >> v1:172.21.15.64:6790/0 conn(0x7f5ccc0a9750 legacy=0x7f5ccc1df3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 >> v1:172.21.15.64:6789/0 conn(0x7f5ccc071c30 legacy=0x7f5ccc126650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 monclient: found mon.b 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc12fed0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 10 monclient: _renew_subs 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc126d60 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 10 monclient: _renew_subs 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc12dd30 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.670+0000 7f5cd0c74640 1 librados: init done 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cc4002040 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5cc4003dc0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc2ffd640 10 monclient: dump: 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:22.666 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.671+0000 7f5cc8d99640 4 set_mon_vals no callback set 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.672+0000 7f5cd0c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.672+0000 7f5cd0c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.672+0000 7f5cd0c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.672+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb0005080 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.673+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f5cc4019d90 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.674+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f5cc40537b0 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.674+0000 7f5cc2ffd640 10 client.4324.objecter ms_dispatch 0x7f5ccc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.674+0000 7f5cc2ffd640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.674+0000 7f5cc2ffd640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.674+0000 7f5cca59c640 10 monclient: get_auth_request con 0x7f5cac06e1e0 auth_method 0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.674+0000 7f5cc2ffd640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.676+0000 7f5cc2ffd640 10 client.4324.objecter ms_handle_connect 0x7f5cac06e1e0 2024-03-29T11:26:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.683+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5cc404fa50 con 0x7f5ccc13e1e0 2024-03-29T11:26:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.683+0000 7f5cc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:22.683+0000 7f5cc2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:23.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.076+0000 7f5cd0c74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.076+0000 7f5cd0c74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.076+0000 7f5cd0c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:23.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.076+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cb0005330 con 0x7f5ccc13e1e0 2024-03-29T11:26:23.065 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:23.065 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:23.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.078+0000 7f5cc2ffd640 1 -- 172.21.15.64:0/3247542291 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f5cc404fbf0 con 0x7f5ccc13e1e0 2024-03-29T11:26:23.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.078+0000 7f5cc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.078+0000 7f5cc2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:23.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 10 librados: watch_flush enter 2024-03-29T11:26:23.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 10 librados: watch_flush exit 2024-03-29T11:26:23.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T11:26:23.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 10 client.4324.objecter successfully canceled tick 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 >> v1:172.21.15.136:6816/2312274743 conn(0x7f5cac06e1e0 legacy=0x7f5cac0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 10 monclient: shutdown 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 >> v1:172.21.15.136:6789/0 conn(0x7f5ccc13e1e0 legacy=0x7f5ccc1dfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 shutdown_connections 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 >> 172.21.15.64:0/3247542291 conn(0x7f5ccc06b660 msgr2=0x7f5ccc13d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.082+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 shutdown_connections 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.083+0000 7f5cd0c74640 1 -- 172.21.15.64:0/3247542291 wait complete. 2024-03-29T11:26:23.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:23.083+0000 7f5cd0c74640 1 librados: shutdown 2024-03-29T11:26:23.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:29.891 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:30.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.043+0000 7fe8d6ee2640 10 monclient: get_monmap_and_config 2024-03-29T11:26:30.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.043+0000 7fe8d6ee2640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.043+0000 7fe8d6ee2640 10 monclient: monmap: 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:30.044245+0000 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:30.044245+0000 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:30.030 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.044+0000 7fe8d6ee2640 1 Processor -- start 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.044+0000 7fe8d6ee2640 1 -- start start 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.044+0000 7fe8d6ee2640 10 monclient: init 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d01422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d0142690 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d00b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d0142c80 con 0x7fe8d0142690 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00aae40 con 0x7fe8d00b17d0 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d013fbd0 con 0x7fe8d01422b0 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d6ee2640 10 monclient(hunting): authenticate will time out at 1326.315225s 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d4c7e640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe8d01422b0 0x7fe8d013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38918/0 (socket says 172.21.15.64:38918) 2024-03-29T11:26:30.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.045+0000 7fe8d4c7e640 1 -- 172.21.15.64:0/3557603321 learned_addr learned my addr 172.21.15.64:0/3557603321 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717981858 0 0) 0x7fe8d00aae40 con 0x7fe8d00b17d0 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0003520 con 0x7fe8d00b17d0 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332028899 0 0) 0x7fe8d013fbd0 con 0x7fe8d01422b0 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d00aae40 con 0x7fe8d01422b0 2024-03-29T11:26:30.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483891846 0 0) 0x7fe8d00aae40 con 0x7fe8d01422b0 2024-03-29T11:26:30.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 >> v1:172.21.15.136:6789/0 conn(0x7fe8d00b17d0 legacy=0x7fe8d00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.046+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 >> v1:172.21.15.64:6789/0 conn(0x7fe8d0142690 legacy=0x7fe8d00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:30.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0151940 con 0x7fe8d01422b0 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe8c4001670 con 0x7fe8d01422b0 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.049+0000 7fe8cf7fe640 10 monclient: dump: 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8d6ee2640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8d6ee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c4001850 con 0x7fe8d01422b0 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3557603321 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe8c4001c20 con 0x7fe8d01422b0 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.050+0000 7fe8cf7fe640 10 monclient: dump: 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:30.041 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.051+0000 7fe8d6ee2640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.051+0000 7fe8d6ee2640 4 set_mon_vals no callback set 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 10 monclient: shutdown 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/3557603321 >> v1:172.21.15.64:6790/0 conn(0x7fe8d01422b0 legacy=0x7fe8d013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/3557603321 shutdown_connections 2024-03-29T11:26:30.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/3557603321 >> 172.21.15.64:0/3557603321 conn(0x7fe8d0066a00 msgr2=0x7fe8d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/3557603321 shutdown_connections 2024-03-29T11:26:30.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.052+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/3557603321 wait complete. 2024-03-29T11:26:30.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.053+0000 7fe8d6ee2640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.054+0000 7fe8d6ee2640 10 monclient: monmap: 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 1 librados: starting msgr at 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 1 librados: starting objecter 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 1 Processor -- start 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 1 -- start start 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 1 librados: setting wanted keys 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 1 librados: calling monclient init 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.055+0000 7fe8d6ee2640 10 monclient: init 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d00b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:30.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d01422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8cffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe8d01422b0 0x7fe8d012fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39118/0 (socket says 172.21.15.64:39118) 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8cffff640 1 -- 172.21.15.64:0/13872138 learned_addr learned my addr 172.21.15.64:0/13872138 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8cd7fa640 10 client.?.objecter ms_handle_connect 0x7fe8d01422b0 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d0142690 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00acb60 con 0x7fe8d0142690 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00ace40 con 0x7fe8d00b17d0 2024-03-29T11:26:30.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8cd7fa640 10 client.?.objecter ms_handle_connect 0x7fe8d00b17d0 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00ad120 con 0x7fe8d01422b0 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.056+0000 7fe8d6ee2640 10 monclient(hunting): authenticate will time out at 1326.326849s 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048315926 0 0) 0x7fe8d00ad120 con 0x7fe8d01422b0 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d01422b0 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675476730 0 0) 0x7fe8d00ace40 con 0x7fe8d00b17d0 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d00ad120 con 0x7fe8d00b17d0 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1789239569 0 0) 0x7fe8b4004eb0 con 0x7fe8d01422b0 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d00ace40 con 0x7fe8d01422b0 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe8bc0018f0 con 0x7fe8d01422b0 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.050 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.057+0000 7fe8cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1605870131 0 0) 0x7fe8d00ad120 con 0x7fe8d00b17d0 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d00b17d0 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe8c4001ba0 con 0x7fe8d00b17d0 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1228549396 0 0) 0x7fe8d00ace40 con 0x7fe8d01422b0 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 >> v1:172.21.15.136:6789/0 conn(0x7fe8d00b17d0 legacy=0x7fe8d00ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.058+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 >> v1:172.21.15.64:6789/0 conn(0x7fe8d0142690 legacy=0x7fe8d0130270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 1 monclient: found mon.c 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0132210 con 0x7fe8d01422b0 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8d6ee2640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:26:30.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8d6ee2640 10 monclient: _renew_subs 2024-03-29T11:26:30.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8d6ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.060+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d00abec0 con 0x7fe8d01422b0 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8d6ee2640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8d6ee2640 10 monclient: _renew_subs 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8d6ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8bc003610 con 0x7fe8d01422b0 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d01e2e70 con 0x7fe8d01422b0 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8d6ee2640 1 librados: init done 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe8bc003a20 con 0x7fe8d01422b0 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8cd7fa640 10 monclient: dump: 2024-03-29T11:26:30.057 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.061+0000 7fe8ce7fc640 4 set_mon_vals no callback set 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.063+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fe8bc003b80 con 0x7fe8d01422b0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8d6ee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8d6ee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8d6ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d0132450 con 0x7fe8d01422b0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fe8bc084a10 con 0x7fe8d01422b0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8cd7fa640 10 client.4331.objecter ms_dispatch 0x7fe8d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8cd7fa640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8cd7fa640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.064+0000 7fe8cd7fa640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.065+0000 7fe8d4c7e640 10 monclient: get_auth_request con 0x7fe8b406de80 auth_method 0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.065+0000 7fe8cd7fa640 10 client.4331.objecter ms_handle_connect 0x7fe8b406de80 2024-03-29T11:26:30.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.071+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8d0132450 con 0x7fe8d01422b0 2024-03-29T11:26:30.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.071+0000 7fe8cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.071+0000 7fe8cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:30.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.389+0000 7fe8d6ee2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.389+0000 7fe8d6ee2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.389+0000 7fe8d6ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:30.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.389+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8d0132450 con 0x7fe8d01422b0 2024-03-29T11:26:30.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.390+0000 7fe8cd7fa640 1 -- 172.21.15.64:0/13872138 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fe8d0132450 con 0x7fe8d01422b0 2024-03-29T11:26:30.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.390+0000 7fe8cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.390+0000 7fe8cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:30.376 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:30.377 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 10 librados: watch_flush enter 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 10 librados: watch_flush exit 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 >> v1:172.21.15.136:6816/2312274743 conn(0x7fe8b406de80 legacy=0x7fe8b4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 10 monclient: shutdown 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 >> v1:172.21.15.64:6790/0 conn(0x7fe8d01422b0 legacy=0x7fe8d012fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 shutdown_connections 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.393+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 >> 172.21.15.64:0/13872138 conn(0x7fe8d0066a00 msgr2=0x7fe8d013c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.394+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 shutdown_connections 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.394+0000 7fe8d6ee2640 1 -- 172.21.15.64:0/13872138 wait complete. 2024-03-29T11:26:30.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:30.394+0000 7fe8d6ee2640 1 librados: shutdown 2024-03-29T11:26:30.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:37.203 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.353+0000 7fdd786a1640 10 monclient: get_monmap_and_config 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.353+0000 7fdd786a1640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.353+0000 7fdd786a1640 10 monclient: monmap: 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:37.354593+0000 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:37.354593+0000 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:37.340 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:37.341 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:37.341 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:37.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.354+0000 7fdd786a1640 1 Processor -- start 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.354+0000 7fdd786a1640 1 -- start start 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.354+0000 7fdd786a1640 10 monclient: init 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient(hunting): picked mon.noname-a con 0x7fdd701457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient(hunting): picked mon.noname-b con 0x7fdd7013f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient(hunting): picked mon.noname-c con 0x7fdd700a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd700aaef0 con 0x7fdd701457f0 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd75c3c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdd7013f650 0x7fdd700b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:44408/0 (socket says 172.21.15.64:44408) 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd75c3c640 1 -- 172.21.15.64:0/2112239702 learned_addr learned my addr 172.21.15.64:0/2112239702 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70145bd0 con 0x7fdd700a9d70 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70145eb0 con 0x7fdd7013f650 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd786a1640 10 monclient(hunting): authenticate will time out at 1333.625550s 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017553611 0 0) 0x7fdd70145eb0 con 0x7fdd7013f650 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd7543b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd7543b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd7543b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd7543b640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.355+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5c003520 con 0x7fdd7013f650 2024-03-29T11:26:37.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169001653 0 0) 0x7fdd70145bd0 con 0x7fdd700a9d70 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd70145eb0 con 0x7fdd700a9d70 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 7198211 0 0) 0x7fdd5c003520 con 0x7fdd7013f650 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 >> v1:172.21.15.136:6789/0 conn(0x7fdd700a9d70 legacy=0x7fdd7014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.356+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 >> v1:172.21.15.64:6789/0 conn(0x7fdd701457f0 legacy=0x7fdd7013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 1 monclient: found mon.noname-b 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70146340 con 0x7fdd7013f650 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdd60001690 con 0x7fdd7013f650 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: dump: 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd786a1640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:26:37.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd786a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd600017f0 con 0x7fdd7013f650 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 1 -- 172.21.15.64:0/2112239702 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdd60001c00 con 0x7fdd7013f650 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd786a1640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd786a1640 4 set_mon_vals no callback set 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.358+0000 7fdd7543b640 10 monclient: dump: 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:37.347 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:37.348 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:37.348 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:37.348 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.359+0000 7fdd786a1640 10 monclient: shutdown 2024-03-29T11:26:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.359+0000 7fdd786a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.359+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 >> v1:172.21.15.64:6790/0 conn(0x7fdd7013f650 legacy=0x7fdd700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.359+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 shutdown_connections 2024-03-29T11:26:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.359+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 >> 172.21.15.64:0/2112239702 conn(0x7fdd70066a00 msgr2=0x7fdd70066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.360+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 shutdown_connections 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.360+0000 7fdd786a1640 1 -- 172.21.15.64:0/2112239702 wait complete. 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.361+0000 7fdd786a1640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.361+0000 7fdd786a1640 10 monclient: monmap: 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 1 librados: starting msgr at 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 1 librados: starting objecter 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 1 Processor -- start 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 1 -- start start 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 1 librados: setting wanted keys 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 1 librados: calling monclient init 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.362+0000 7fdd786a1640 10 monclient: init 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient(hunting): picked mon.noname-c con 0x7fdd700a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient(hunting): picked mon.noname-a con 0x7fdd7013f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient(hunting): picked mon.noname-b con 0x7fdd701457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70142e70 con 0x7fdd7013f650 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70141c80 con 0x7fdd701457f0 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70142b00 con 0x7fdd700a9d70 2024-03-29T11:26:37.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.363+0000 7fdd786a1640 10 monclient(hunting): authenticate will time out at 1333.633796s 2024-03-29T11:26:37.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.364+0000 7fdd7643d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdd700a9d70 0x7fdd701423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47348/0 (socket says 172.21.15.64:47348) 2024-03-29T11:26:37.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.364+0000 7fdd7643d640 1 -- 172.21.15.64:0/678680002 learned_addr learned my addr 172.21.15.64:0/678680002 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:37.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 10 client.?.objecter ms_handle_connect 0x7fdd700a9d70 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456331370 0 0) 0x7fdd70142b00 con 0x7fdd700a9d70 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.365+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54004eb0 con 0x7fdd700a9d70 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2033747953 0 0) 0x7fdd54004eb0 con 0x7fdd700a9d70 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd70142b00 con 0x7fdd700a9d70 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdd6c003fe0 con 0x7fdd700a9d70 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:37.360 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 client.?.objecter ms_handle_connect 0x7fdd701457f0 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386944126 0 0) 0x7fdd70141c80 con 0x7fdd701457f0 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54004eb0 con 0x7fdd701457f0 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 client.?.objecter ms_handle_connect 0x7fdd7013f650 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.366+0000 7fdd66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156032467 0 0) 0x7fdd70142e70 con 0x7fdd7013f650 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd70141c80 con 0x7fdd7013f650 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 184495890 0 0) 0x7fdd70142b00 con 0x7fdd700a9d70 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 >> v1:172.21.15.136:6789/0 conn(0x7fdd701457f0 legacy=0x7fdd701daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 >> v1:172.21.15.64:6789/0 conn(0x7fdd7013f650 legacy=0x7fdd70134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 monclient: found mon.c 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7013e300 con 0x7fdd700a9d70 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd786a1640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd786a1640 10 monclient: _renew_subs 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd786a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7013e5b0 con 0x7fdd700a9d70 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd6c000ed0 con 0x7fdd700a9d70 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd67fff640 4 set_mon_vals no callback set 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.367+0000 7fdd786a1640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd786a1640 10 monclient: _renew_subs 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd786a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd70135590 con 0x7fdd700a9d70 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd786a1640 1 librados: init done 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdd6c001260 con 0x7fdd700a9d70 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.368+0000 7fdd66ffd640 10 monclient: dump: 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.364 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.369+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fdd6c001440 con 0x7fdd700a9d70 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.369+0000 7fdd786a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.369+0000 7fdd786a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.369+0000 7fdd786a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.369+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd70142c90 con 0x7fdd700a9d70 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fdd6c084af0 con 0x7fdd700a9d70 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd66ffd640 10 client.4337.objecter ms_dispatch 0x7fdd70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd66ffd640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd66ffd640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:37.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd66ffd640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:26:37.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd75c3c640 10 monclient: get_auth_request con 0x7fdd5406de20 auth_method 0 2024-03-29T11:26:37.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.370+0000 7fdd66ffd640 10 client.4337.objecter ms_handle_connect 0x7fdd5406de20 2024-03-29T11:26:37.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.376+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd6c0503d0 con 0x7fdd700a9d70 2024-03-29T11:26:37.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.376+0000 7fdd66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.376+0000 7fdd66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:37.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.683+0000 7fdd786a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.683+0000 7fdd786a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.683+0000 7fdd786a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:37.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.683+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd70075f10 con 0x7fdd700a9d70 2024-03-29T11:26:37.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.684+0000 7fdd66ffd640 1 -- 172.21.15.64:0/678680002 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fdd6c052300 con 0x7fdd700a9d70 2024-03-29T11:26:37.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.684+0000 7fdd66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.684+0000 7fdd66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:37.671 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:37.671 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:37.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.687+0000 7fdd786a1640 10 librados: watch_flush enter 2024-03-29T11:26:37.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.687+0000 7fdd786a1640 10 librados: watch_flush exit 2024-03-29T11:26:37.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.687+0000 7fdd786a1640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:26:37.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.687+0000 7fdd786a1640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:26:37.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 >> v1:172.21.15.136:6816/2312274743 conn(0x7fdd5406de20 legacy=0x7fdd540702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 10 monclient: shutdown 2024-03-29T11:26:37.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 >> v1:172.21.15.64:6790/0 conn(0x7fdd700a9d70 legacy=0x7fdd701423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 shutdown_connections 2024-03-29T11:26:37.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 >> 172.21.15.64:0/678680002 conn(0x7fdd70066a00 msgr2=0x7fdd700b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 shutdown_connections 2024-03-29T11:26:37.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 -- 172.21.15.64:0/678680002 wait complete. 2024-03-29T11:26:37.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:37.688+0000 7fdd786a1640 1 librados: shutdown 2024-03-29T11:26:37.691 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:44.499 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.660+0000 7fac96f87640 10 monclient: get_monmap_and_config 2024-03-29T11:26:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.660+0000 7fac96f87640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.660+0000 7fac96f87640 10 monclient: monmap: 2024-03-29T11:26:44.647 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:44.661567+0000 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:44.661567+0000 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:44.648 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.661+0000 7fac96f87640 1 Processor -- start 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.661+0000 7fac96f87640 1 -- start start 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.661+0000 7fac96f87640 10 monclient: init 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient(hunting): picked mon.noname-a con 0x7fac880389e0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient(hunting): picked mon.noname-b con 0x7fac88038dc0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient(hunting): picked mon.noname-c con 0x7fac88030660 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac880c54c0 con 0x7fac880389e0 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac88036430 con 0x7fac88030660 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac88039840 con 0x7fac88038dc0 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.662+0000 7fac96f87640 10 monclient(hunting): authenticate will time out at 1340.932511s 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac96786640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fac88030660 0x7fac880d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:38496/0 (socket says 172.21.15.64:38496) 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac96786640 1 -- 172.21.15.64:0/1975942263 learned_addr learned my addr 172.21.15.64:0/1975942263 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470584063 0 0) 0x7fac88036430 con 0x7fac88030660 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac94f83640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac94f83640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac94f83640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.663+0000 7fac94f83640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.664+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c003520 con 0x7fac88030660 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.664+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1794358141 0 0) 0x7fac7c003520 con 0x7fac88030660 2024-03-29T11:26:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.664+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 >> v1:172.21.15.64:6790/0 conn(0x7fac88038dc0 legacy=0x7fac880cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 >> v1:172.21.15.64:6789/0 conn(0x7fac880389e0 legacy=0x7fac88033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 1 monclient: found mon.noname-c 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88035690 con 0x7fac88030660 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.665+0000 7fac94f83640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac94f83640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac94f83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac96f87640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:26:44.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac96f87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fac8c001690 con 0x7fac88030660 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac94f83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac94f83640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac94f83640 10 monclient: dump: 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:44.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.666+0000 7fac96f87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:44.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.668+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac8c001860 con 0x7fac88030660 2024-03-29T11:26:44.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.668+0000 7fac94f83640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.668+0000 7fac96f87640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.668+0000 7fac96f87640 4 set_mon_vals no callback set 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac94f83640 1 -- 172.21.15.64:0/1975942263 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fac8c001cb0 con 0x7fac88030660 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac94f83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac94f83640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac94f83640 10 monclient: dump: 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:44.658 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:44.659 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:44.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac96f87640 10 monclient: shutdown 2024-03-29T11:26:44.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac96f87640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.669+0000 7fac96f87640 1 -- 172.21.15.64:0/1975942263 >> v1:172.21.15.136:6789/0 conn(0x7fac88030660 legacy=0x7fac880d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.670+0000 7fac96f87640 1 -- 172.21.15.64:0/1975942263 shutdown_connections 2024-03-29T11:26:44.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.670+0000 7fac96f87640 1 -- 172.21.15.64:0/1975942263 >> 172.21.15.64:0/1975942263 conn(0x7fac88005c00 msgr2=0x7fac88006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.671+0000 7fac96f87640 1 -- 172.21.15.64:0/1975942263 shutdown_connections 2024-03-29T11:26:44.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.671+0000 7fac96f87640 1 -- 172.21.15.64:0/1975942263 wait complete. 2024-03-29T11:26:44.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 10 monclient: monmap: 2024-03-29T11:26:44.663 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:44.663 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 1 librados: starting msgr at 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 1 librados: starting objecter 2024-03-29T11:26:44.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 1 Processor -- start 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 1 -- start start 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 1 librados: setting wanted keys 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 1 librados: calling monclient init 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.672+0000 7fac96f87640 10 monclient: init 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient(hunting): picked mon.noname-a con 0x7fac88030660 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient(hunting): picked mon.noname-b con 0x7fac880389e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient(hunting): picked mon.noname-c con 0x7fac880c7280 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac880c8060 con 0x7fac88030660 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac880c2bd0 con 0x7fac880389e0 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac880c2d60 con 0x7fac880c7280 2024-03-29T11:26:44.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.673+0000 7fac96f87640 10 monclient(hunting): authenticate will time out at 1340.943704s 2024-03-29T11:26:44.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.676+0000 7fac95784640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fac880389e0 0x7fac880c24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:43342/0 (socket says 172.21.15.64:43342) 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.676+0000 7fac95784640 1 -- 172.21.15.64:0/2160544906 learned_addr learned my addr 172.21.15.64:0/2160544906 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 client.?.objecter ms_handle_connect 0x7fac88030660 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 client.?.objecter ms_handle_connect 0x7fac880c7280 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542832224 0 0) 0x7fac880c8060 con 0x7fac88030660 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac74004eb0 con 0x7fac88030660 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 client.?.objecter ms_handle_connect 0x7fac880389e0 2024-03-29T11:26:44.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.677+0000 7fac867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1914677794 0 0) 0x7fac74004eb0 con 0x7fac88030660 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac74005300 con 0x7fac88030660 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032523317 0 0) 0x7fac880c2d60 con 0x7fac880c7280 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac74005760 con 0x7fac880c7280 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fac9004fdd0 con 0x7fac88030660 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:44.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): dump: 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986985979 0 0) 0x7fac880c2bd0 con 0x7fac880389e0 2024-03-29T11:26:44.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac880c2d60 con 0x7fac880389e0 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2328104488 0 0) 0x7fac74005300 con 0x7fac88030660 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 >> v1:172.21.15.64:6790/0 conn(0x7fac880c7280 legacy=0x7fac8815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 >> v1:172.21.15.136:6789/0 conn(0x7fac880389e0 legacy=0x7fac880c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 monclient: found mon.a 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac880c2ef0 con 0x7fac88030660 2024-03-29T11:26:44.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac96f87640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac96f87640 10 monclient: _renew_subs 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac96f87640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.678+0000 7fac96f87640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8802f0b0 con 0x7fac88030660 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.679+0000 7fac96f87640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.679+0000 7fac96f87640 10 monclient: _renew_subs 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.679+0000 7fac96f87640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.679+0000 7fac96f87640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8815d210 con 0x7fac88030660 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.679+0000 7fac96f87640 1 librados: init done 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac7bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac7bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac9004f400 con 0x7fac88030660 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac88007b90 con 0x7fac88030660 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fac9004fdd0 con 0x7fac88030660 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.682+0000 7fac867fc640 10 monclient: dump: 2024-03-29T11:26:44.676 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:44.677 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.683+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fac90067bf0 con 0x7fac88030660 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.684+0000 7fac877fe640 4 set_mon_vals no callback set 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fac900cdd50 con 0x7fac88030660 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac867fc640 10 client.4320.objecter ms_dispatch 0x7fac90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac867fc640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac867fc640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac867fc640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac95784640 10 monclient: get_auth_request con 0x7fac7406df60 auth_method 0 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.685+0000 7fac867fc640 10 client.4320.objecter ms_handle_connect 0x7fac7406df60 2024-03-29T11:26:44.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.691+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac900c9ff0 con 0x7fac88030660 2024-03-29T11:26:44.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.691+0000 7fac867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:44.691+0000 7fac867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:45.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.021+0000 7fac7bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.021+0000 7fac7bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.021+0000 7fac7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:26:45.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.021+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac88005120 con 0x7fac88030660 2024-03-29T11:26:45.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.022+0000 7fac867fc640 1 -- 172.21.15.64:0/2160544906 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fac900c9ff0 con 0x7fac88030660 2024-03-29T11:26:45.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.022+0000 7fac867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.022+0000 7fac867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:45.010 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:45.010 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:45.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 10 librados: watch_flush enter 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 10 librados: watch_flush exit 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 10 client.4320.objecter successfully canceled tick 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 >> v1:172.21.15.136:6816/2312274743 conn(0x7fac7406df60 legacy=0x7fac74070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 10 monclient: shutdown 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 >> v1:172.21.15.64:6789/0 conn(0x7fac88030660 legacy=0x7fac880c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 shutdown_connections 2024-03-29T11:26:45.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.027+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 >> 172.21.15.64:0/2160544906 conn(0x7fac88005c00 msgr2=0x7fac88034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:45.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.029+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 shutdown_connections 2024-03-29T11:26:45.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.029+0000 7fac7bfff640 1 -- 172.21.15.64:0/2160544906 wait complete. 2024-03-29T11:26:45.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:45.029+0000 7fac7bfff640 1 librados: shutdown 2024-03-29T11:26:45.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:51.843 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.003+0000 7fd6f3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.003+0000 7fd6f3fff640 10 monclient: build_initial_monmap 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.003+0000 7fd6f3fff640 10 monclient: monmap: 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:52.005807+0000 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:52.005807+0000 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:51.992 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:51.993 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:51.993 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:51.993 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:51.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.005+0000 7fd6f3fff640 1 Processor -- start 2024-03-29T11:26:51.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.005+0000 7fd6f3fff640 1 -- start start 2024-03-29T11:26:51.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.005+0000 7fd6f3fff640 10 monclient: init 2024-03-29T11:26:51.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:51.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:51.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd6ec02b4c0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:51.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd6ec00b8a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:51.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd6ec02a5a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:51.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec02c5c0 con 0x7fd6ec02a5a0 2024-03-29T11:26:51.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec0c11a0 con 0x7fd6ec00b8a0 2024-03-29T11:26:51.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec02afa0 con 0x7fd6ec02b4c0 2024-03-29T11:26:51.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:51.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.006+0000 7fd6f3fff640 10 monclient(hunting): authenticate will time out at 1348.276979s 2024-03-29T11:26:51.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.008+0000 7fd6f37fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd6ec02a5a0 0x7fd6ec0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55211/0 (socket says 172.21.15.64:55211) 2024-03-29T11:26:51.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.008+0000 7fd6f37fe640 1 -- 172.21.15.64:0/45279754 learned_addr learned my addr 172.21.15.64:0/45279754 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196437419 0 0) 0x7fd6ec02c5c0 con 0x7fd6ec02a5a0 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6dc003520 con 0x7fd6ec02a5a0 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1957709073 0 0) 0x7fd6dc003520 con 0x7fd6ec02a5a0 2024-03-29T11:26:51.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.009+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 >> v1:172.21.15.64:6790/0 conn(0x7fd6ec02b4c0 legacy=0x7fd6ec006890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:52.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 >> v1:172.21.15.136:6789/0 conn(0x7fd6ec00b8a0 legacy=0x7fd6ec0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec0cb160 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f3fff640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd6e8001690 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: dump: 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:52.003 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6e80018b0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.014+0000 7fd6f3fff640 4 set_mon_vals no callback set 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f1ffb640 1 -- 172.21.15.64:0/45279754 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd6e8001c40 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f1ffb640 10 monclient: dump: 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:52.005 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:52.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 10 monclient: shutdown 2024-03-29T11:26:52.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 1 -- 172.21.15.64:0/45279754 >> v1:172.21.15.64:6789/0 conn(0x7fd6ec02a5a0 legacy=0x7fd6ec0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 1 -- 172.21.15.64:0/45279754 shutdown_connections 2024-03-29T11:26:52.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 1 -- 172.21.15.64:0/45279754 >> 172.21.15.64:0/45279754 conn(0x7fd6ec0058f0 msgr2=0x7fd6ec005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 1 -- 172.21.15.64:0/45279754 shutdown_connections 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.015+0000 7fd6f3fff640 1 -- 172.21.15.64:0/45279754 wait complete. 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.021+0000 7fd6f3fff640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.021+0000 7fd6f3fff640 10 monclient: monmap: 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:52.010 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:52.011 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:52.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.021+0000 7fd6f3fff640 1 librados: starting msgr at 2024-03-29T11:26:52.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.021+0000 7fd6f3fff640 1 librados: starting objecter 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.021+0000 7fd6f3fff640 1 Processor -- start 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.021+0000 7fd6f3fff640 1 -- start start 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 1 librados: setting wanted keys 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 1 librados: calling monclient init 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient: init 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd6ec00b8a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd6ec02a5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd6ec02b4c0 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:52.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec0b41e0 con 0x7fd6ec00b8a0 2024-03-29T11:26:52.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec0b6aa0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec0c3440 con 0x7fd6ec02b4c0 2024-03-29T11:26:52.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.022+0000 7fd6f3fff640 10 monclient(hunting): authenticate will time out at 1348.293690s 2024-03-29T11:26:52.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6f27fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd6ec02a5a0 0x7fd6ec0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:44835/0 (socket says 172.21.15.64:44835) 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6f27fc640 1 -- 172.21.15.64:0/3433549767 learned_addr learned my addr 172.21.15.64:0/3433549767 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 10 client.?.objecter ms_handle_connect 0x7fd6ec02a5a0 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529016693 0 0) 0x7fd6ec0b6aa0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.026+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e0004eb0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 375831913 0 0) 0x7fd6e0004eb0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6ec0b6aa0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd6f404fdd0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 10 client.?.objecter ms_handle_connect 0x7fd6ec00b8a0 2024-03-29T11:26:52.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2569290882 0 0) 0x7fd6ec0b6aa0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.027+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 >> v1:172.21.15.64:6790/0 conn(0x7fd6ec02b4c0 legacy=0x7fd6ec0b3850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 >> v1:172.21.15.64:6789/0 conn(0x7fd6ec00b8a0 legacy=0x7fd6ec0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 1 monclient: found mon.b 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec0b3f60 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 10 monclient: _renew_subs 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:52.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec0c2630 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 10 monclient: _renew_subs 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec0d3410 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.028+0000 7fd6f3fff640 1 librados: init done 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6f404f400 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd6f4075710 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6db7fe640 10 monclient: dump: 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:52.024 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.030+0000 7fd6f17fa640 4 set_mon_vals no callback set 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.032+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fd6f4067bf0 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.032+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fd6f40ff550 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.032+0000 7fd6db7fe640 10 client.4348.objecter ms_dispatch 0x7fd6f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.032+0000 7fd6db7fe640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.032+0000 7fd6db7fe640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.033+0000 7fd6db7fe640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:26:52.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.033+0000 7fd6f3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.033+0000 7fd6f3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.033+0000 7fd6f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:52.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.033+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d4005080 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.036+0000 7fd6f2ffd640 10 monclient: get_auth_request con 0x7fd6e006d9e0 auth_method 0 2024-03-29T11:26:52.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.037+0000 7fd6db7fe640 10 client.4348.objecter ms_handle_connect 0x7fd6e006d9e0 2024-03-29T11:26:52.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.042+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6f40cae30 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.042+0000 7fd6db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.042+0000 7fd6db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:52.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.376+0000 7fd6f3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.376+0000 7fd6f3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.376+0000 7fd6f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:26:52.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.376+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6d4005330 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.378+0000 7fd6db7fe640 1 -- 172.21.15.64:0/3433549767 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fd6f40ccd60 con 0x7fd6ec02a5a0 2024-03-29T11:26:52.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.378+0000 7fd6db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.378+0000 7fd6db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:52.367 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:52.367 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:52.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.382+0000 7fd6f3fff640 10 librados: watch_flush enter 2024-03-29T11:26:52.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.382+0000 7fd6f3fff640 10 librados: watch_flush exit 2024-03-29T11:26:52.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.382+0000 7fd6f3fff640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:26:52.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.382+0000 7fd6f3fff640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:26:52.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 >> v1:172.21.15.136:6816/2312274743 conn(0x7fd6e006d9e0 legacy=0x7fd6e006fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 10 monclient: shutdown 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 >> v1:172.21.15.136:6789/0 conn(0x7fd6ec02a5a0 legacy=0x7fd6ec0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 shutdown_connections 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 >> 172.21.15.64:0/3433549767 conn(0x7fd6ec0058f0 msgr2=0x7fd6ec0334d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 shutdown_connections 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.385+0000 7fd6f3fff640 1 -- 172.21.15.64:0/3433549767 wait complete. 2024-03-29T11:26:52.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:52.386+0000 7fd6f3fff640 1 librados: shutdown 2024-03-29T11:26:52.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:59.201 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.365+0000 7f84e6bba640 10 monclient: get_monmap_and_config 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.365+0000 7f84e6bba640 10 monclient: build_initial_monmap 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.365+0000 7f84e6bba640 10 monclient: monmap: 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:26:59.366918+0000 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:26:59.366918+0000 2024-03-29T11:26:59.353 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:59.354 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:59.354 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:59.354 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:26:59.354 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:26:59.354 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.367+0000 7f84e6bba640 1 Processor -- start 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.367+0000 7f84e6bba640 1 -- start start 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.367+0000 7f84e6bba640 10 monclient: init 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.367+0000 7f84e6bba640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.367+0000 7f84e6bba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.367+0000 7f84e6bba640 10 monclient(hunting): picked mon.noname-b con 0x7f84e00b4890 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 10 monclient(hunting): picked mon.noname-c con 0x7f84e013c040 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 10 monclient(hunting): picked mon.noname-a con 0x7f84e01427e0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00b42f0 con 0x7f84e01427e0 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00aa7a0 con 0x7f84e013c040 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00aaa10 con 0x7f84e00b4890 2024-03-29T11:26:59.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:59.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.368+0000 7f84e6bba640 10 monclient(hunting): authenticate will time out at 1355.637957s 2024-03-29T11:26:59.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.371+0000 7f84e4956640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f84e00b4890 0x7f84e013b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46469/0 (socket says 172.21.15.64:46469) 2024-03-29T11:26:59.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.371+0000 7f84e4956640 1 -- 172.21.15.64:0/3510532072 learned_addr learned my addr 172.21.15.64:0/3510532072 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:59.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.372+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611823388 0 0) 0x7f84e00b42f0 con 0x7f84e01427e0 2024-03-29T11:26:59.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.372+0000 7f84df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.372+0000 7f84df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.372+0000 7f84df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.372+0000 7f84df7fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:26:59.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.372+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d0003520 con 0x7f84e01427e0 2024-03-29T11:26:59.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.373+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092920418 0 0) 0x7f84d0003520 con 0x7f84e01427e0 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.373+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 >> v1:172.21.15.64:6790/0 conn(0x7f84e00b4890 legacy=0x7f84e013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.373+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 >> v1:172.21.15.136:6789/0 conn(0x7f84e013c040 legacy=0x7f84e01420d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.373+0000 7f84df7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.373+0000 7f84df7fe640 1 monclient: found mon.noname-a 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00aacf0 con 0x7f84e01427e0 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84e6bba640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84e6bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:59.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f84d8001a10 con 0x7f84e01427e0 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84df7fe640 10 monclient: dump: 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:59.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.374+0000 7f84e6bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:59.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d8001c30 con 0x7f84e01427e0 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84e6bba640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84e6bba640 4 set_mon_vals no callback set 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84df7fe640 1 -- 172.21.15.64:0/3510532072 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f84d8001fc0 con 0x7f84e01427e0 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84df7fe640 10 monclient: dump: 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.366 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84e6bba640 10 monclient: shutdown 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84e6bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.378+0000 7f84e6bba640 1 -- 172.21.15.64:0/3510532072 >> v1:172.21.15.64:6789/0 conn(0x7f84e01427e0 legacy=0x7f84e014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.379+0000 7f84e6bba640 1 -- 172.21.15.64:0/3510532072 shutdown_connections 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.379+0000 7f84e6bba640 1 -- 172.21.15.64:0/3510532072 >> 172.21.15.64:0/3510532072 conn(0x7f84e006d180 msgr2=0x7f84e0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.379+0000 7f84e6bba640 1 -- 172.21.15.64:0/3510532072 shutdown_connections 2024-03-29T11:26:59.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.379+0000 7f84e6bba640 1 -- 172.21.15.64:0/3510532072 wait complete. 2024-03-29T11:26:59.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.380+0000 7f84e6bba640 10 monclient: build_initial_monmap 2024-03-29T11:26:59.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.380+0000 7f84e6bba640 10 monclient: monmap: 2024-03-29T11:26:59.368 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:26:59.368 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 1 librados: starting msgr at 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 1 librados: starting objecter 2024-03-29T11:26:59.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 1 Processor -- start 2024-03-29T11:26:59.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 1 -- start start 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 1 librados: setting wanted keys 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 1 librados: calling monclient init 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.381+0000 7f84e6bba640 10 monclient: init 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient(hunting): picked mon.noname-b con 0x7f84e00b4890 addr v1:172.21.15.136:6789/0 2024-03-29T11:26:59.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient(hunting): picked mon.noname-c con 0x7f84e013c040 addr v1:172.21.15.64:6790/0 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient(hunting): picked mon.noname-a con 0x7f84e01427e0 addr v1:172.21.15.64:6789/0 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e01d6ff0 con 0x7f84e01427e0 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00acb80 con 0x7f84e00b4890 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e0138710 con 0x7f84e013c040 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.382+0000 7f84e6bba640 10 monclient(hunting): authenticate will time out at 1355.652598s 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.386+0000 7f84dffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f84e013c040 0x7f84e01378f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46573/0 (socket says 172.21.15.64:46573) 2024-03-29T11:26:59.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.386+0000 7f84dffff640 1 -- 172.21.15.64:0/681045013 learned_addr learned my addr 172.21.15.64:0/681045013 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.388+0000 7f84dd7fa640 10 client.?.objecter ms_handle_connect 0x7f84e013c040 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.388+0000 7f84dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.388+0000 7f84dd7fa640 10 client.?.objecter ms_handle_connect 0x7f84e00b4890 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889755632 0 0) 0x7f84e0138710 con 0x7f84e013c040 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c4004eb0 con 0x7f84e013c040 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287911407 0 0) 0x7f84e00acb80 con 0x7f84e00b4890 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.389+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e0138710 con 0x7f84e00b4890 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.390+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3091949744 0 0) 0x7f84c4004eb0 con 0x7f84e013c040 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.390+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84e00acb80 con 0x7f84e013c040 2024-03-29T11:26:59.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.390+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f84cc001870 con 0x7f84e013c040 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.390+0000 7f84dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.381 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2494226664 0 0) 0x7f84e00acb80 con 0x7f84e013c040 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 >> v1:172.21.15.136:6789/0 conn(0x7f84e00b4890 legacy=0x7f84e01d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 >> v1:172.21.15.64:6789/0 conn(0x7f84e01427e0 legacy=0x7f84e0138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 1 monclient: found mon.c 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:59.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e013c590 con 0x7f84e013c040 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.391+0000 7f84dd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 10 monclient: _renew_subs 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e01d7b90 con 0x7f84e013c040 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 10 monclient: _renew_subs 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e013f760 con 0x7f84e013c040 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84e6bba640 1 librados: init done 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84cc0019d0 con 0x7f84e013c040 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:59.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f84cc004aa0 con 0x7f84e013c040 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84dd7fa640 10 monclient: dump: 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:26:59.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.392+0000 7f84deffd640 4 set_mon_vals no callback set 2024-03-29T11:26:59.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.394+0000 7f84e6bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.394+0000 7f84e6bba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.394+0000 7f84e6bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:59.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.394+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c0005080 con 0x7f84e013c040 2024-03-29T11:26:59.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.394+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f84cc004c00 con 0x7f84e013c040 2024-03-29T11:26:59.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f84cc084aa0 con 0x7f84e013c040 2024-03-29T11:26:59.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 10 client.4346.objecter ms_dispatch 0x7f84e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:59.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84cc084ea0 con 0x7f84e013c040 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.403+0000 7f84e4956640 10 monclient: get_auth_request con 0x7f84c406ddc0 auth_method 0 2024-03-29T11:26:59.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.404+0000 7f84dd7fa640 10 client.4346.objecter ms_handle_connect 0x7f84c406ddc0 2024-03-29T11:26:59.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.730+0000 7f84e6bba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.730+0000 7f84e6bba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.730+0000 7f84e6bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:26:59.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.730+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c0005330 con 0x7f84e013c040 2024-03-29T11:26:59.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.734+0000 7f84dd7fa640 1 -- 172.21.15.64:0/681045013 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f84cc050380 con 0x7f84e013c040 2024-03-29T11:26:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.734+0000 7f84dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.734+0000 7f84dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:59.722 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:26:59.722 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:59.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 10 librados: watch_flush enter 2024-03-29T11:26:59.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 10 librados: watch_flush exit 2024-03-29T11:26:59.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:26:59.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:26:59.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 >> v1:172.21.15.136:6816/2312274743 conn(0x7f84c406ddc0 legacy=0x7f84c4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 10 monclient: shutdown 2024-03-29T11:26:59.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.738+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 >> v1:172.21.15.64:6790/0 conn(0x7f84e013c040 legacy=0x7f84e01378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.741+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 shutdown_connections 2024-03-29T11:26:59.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.741+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 >> 172.21.15.64:0/681045013 conn(0x7f84e006d180 msgr2=0x7f84e0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.741+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 shutdown_connections 2024-03-29T11:26:59.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.741+0000 7f84e6bba640 1 -- 172.21.15.64:0/681045013 wait complete. 2024-03-29T11:26:59.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:26:59.741+0000 7f84e6bba640 1 librados: shutdown 2024-03-29T11:26:59.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:06.554 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:06.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.719+0000 7f5fce684640 10 monclient: get_monmap_and_config 2024-03-29T11:27:06.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.719+0000 7f5fce684640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.719+0000 7f5fce684640 10 monclient: monmap: 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:06.720557+0000 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:06.720557+0000 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:06.708 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.723+0000 7f5fce684640 1 Processor -- start 2024-03-29T11:27:06.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.723+0000 7f5fce684640 1 -- start start 2024-03-29T11:27:06.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.723+0000 7f5fce684640 10 monclient: init 2024-03-29T11:27:06.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc0034850 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc002a640 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc00327d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc002c820 con 0x7f5fc00327d0 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc002a120 con 0x7f5fc0034850 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00c75b0 con 0x7f5fc002a640 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.724+0000 7f5fce684640 10 monclient(hunting): authenticate will time out at 1362.994766s 2024-03-29T11:27:06.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.729+0000 7f5fcce81640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5fc002a640 0x7f5fc00320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39271/0 (socket says 172.21.15.64:39271) 2024-03-29T11:27:06.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fcce81640 1 -- 172.21.15.64:0/2080077200 learned_addr learned my addr 172.21.15.64:0/2080077200 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:06.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862958593 0 0) 0x7f5fc00c75b0 con 0x7f5fc002a640 2024-03-29T11:27:06.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb4003520 con 0x7f5fc002a640 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805110697 0 0) 0x7f5fb4003520 con 0x7f5fc002a640 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 >> v1:172.21.15.136:6789/0 conn(0x7f5fc0034850 legacy=0x7f5fc00c6ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:27:06.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.730+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 >> v1:172.21.15.64:6789/0 conn(0x7f5fc00327d0 legacy=0x7f5fc00cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:06.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 1 monclient: found mon.noname-b 2024-03-29T11:27:06.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc0033110 con 0x7f5fc002a640 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5fb8001670 con 0x7f5fc002a640 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fbffff640 10 monclient: dump: 2024-03-29T11:27:06.722 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:06.722 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fce684640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.731+0000 7f5fce684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fb80018d0 con 0x7f5fc002a640 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fbffff640 1 -- 172.21.15.64:0/2080077200 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5fb8001d20 con 0x7f5fc002a640 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fbffff640 10 monclient: dump: 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.724 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fce684640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.732+0000 7f5fce684640 4 set_mon_vals no callback set 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.733+0000 7f5fce684640 10 monclient: shutdown 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.733+0000 7f5fce684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.733+0000 7f5fce684640 1 -- 172.21.15.64:0/2080077200 >> v1:172.21.15.64:6790/0 conn(0x7f5fc002a640 legacy=0x7f5fc00320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.736+0000 7f5fce684640 1 -- 172.21.15.64:0/2080077200 shutdown_connections 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.736+0000 7f5fce684640 1 -- 172.21.15.64:0/2080077200 >> 172.21.15.64:0/2080077200 conn(0x7f5fc00058f0 msgr2=0x7f5fc0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.736+0000 7f5fce684640 1 -- 172.21.15.64:0/2080077200 shutdown_connections 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.736+0000 7f5fce684640 1 -- 172.21.15.64:0/2080077200 wait complete. 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.737+0000 7f5fce684640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.737+0000 7f5fce684640 10 monclient: monmap: 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:06.730 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.737+0000 7f5fce684640 1 librados: starting msgr at 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.737+0000 7f5fce684640 1 librados: starting objecter 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.737+0000 7f5fce684640 1 Processor -- start 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.738+0000 7f5fce684640 1 -- start start 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.738+0000 7f5fce684640 1 librados: setting wanted keys 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.738+0000 7f5fce684640 1 librados: calling monclient init 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.738+0000 7f5fce684640 10 monclient: init 2024-03-29T11:27:06.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc002a640 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc00327d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc0034850 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc0169510 con 0x7f5fc002a640 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc002d9a0 con 0x7f5fc00327d0 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc002dc80 con 0x7f5fc0034850 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.739+0000 7f5fce684640 10 monclient(hunting): authenticate will time out at 1363.008987s 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.742+0000 7f5fcd682640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5fc002a640 0x7f5fc0169c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52183/0 (socket says 172.21.15.64:52183) 2024-03-29T11:27:06.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.742+0000 7f5fcd682640 1 -- 172.21.15.64:0/2878678966 learned_addr learned my addr 172.21.15.64:0/2878678966 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:06.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.743+0000 7f5fbdffb640 10 client.?.objecter ms_handle_connect 0x7f5fc002a640 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.743+0000 7f5fbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.745+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467969247 0 0) 0x7f5fc0169510 con 0x7f5fc002a640 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.745+0000 7f5fbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.745+0000 7f5fbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.745+0000 7f5fbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.745+0000 7f5fbdffb640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:27:06.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.745+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac004eb0 con 0x7f5fc002a640 2024-03-29T11:27:06.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.748+0000 7f5fbdffb640 10 client.?.objecter ms_handle_connect 0x7f5fc00327d0 2024-03-29T11:27:06.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.748+0000 7f5fbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.748+0000 7f5fbdffb640 10 client.?.objecter ms_handle_connect 0x7f5fc0034850 2024-03-29T11:27:06.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.748+0000 7f5fbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.748+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837446621 0 0) 0x7f5fc002d9a0 con 0x7f5fc00327d0 2024-03-29T11:27:06.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.748+0000 7f5fbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.749+0000 7f5fbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.749+0000 7f5fbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.749+0000 7f5fbdffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:27:06.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.749+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc0169510 con 0x7f5fc00327d0 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342327290 0 0) 0x7f5fc002dc80 con 0x7f5fc0034850 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc002d9a0 con 0x7f5fc0034850 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 891504762 0 0) 0x7f5fac004eb0 con 0x7f5fc002a640 2024-03-29T11:27:06.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc002dc80 con 0x7f5fc002a640 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5fc804fdd0 con 0x7f5fc002a640 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): dump: 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:06.739 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933454555 0 0) 0x7f5fc002d9a0 con 0x7f5fc0034850 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fac004eb0 con 0x7f5fc0034850 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5fc40016e0 con 0x7f5fc0034850 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 10 monclient(hunting): dump: 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.740 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.741 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.741 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.742 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:06.742 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:06.742 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:06.742 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168785695 0 0) 0x7f5fc0169510 con 0x7f5fc00327d0 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.750+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc002d9a0 con 0x7f5fc00327d0 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5fb8001630 con 0x7f5fc00327d0 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient(hunting): dump: 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.743 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 530538142 0 0) 0x7f5fac004eb0 con 0x7f5fc0034850 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 >> v1:172.21.15.136:6789/0 conn(0x7f5fc00327d0 legacy=0x7f5fc00303f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 >> v1:172.21.15.64:6789/0 conn(0x7f5fc002a640 legacy=0x7f5fc0169c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 1 monclient: found mon.c 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc0169760 con 0x7f5fc0034850 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.751+0000 7f5fbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.752+0000 7f5fce684640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.752+0000 7f5fce684640 10 monclient: _renew_subs 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.752+0000 7f5fce684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.752+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc016a370 con 0x7f5fc0034850 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fce684640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fce684640 10 monclient: _renew_subs 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fce684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc0031370 con 0x7f5fc0034850 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fce684640 1 librados: init done 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fc4001840 con 0x7f5fc0034850 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5fc4001d70 con 0x7f5fc0034850 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbdffb640 10 monclient: dump: 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.753+0000 7f5fbf7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:06.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.755+0000 7f5fce684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.756+0000 7f5fce684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.756+0000 7f5fce684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:06.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.756+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa8005080 con 0x7f5fc0034850 2024-03-29T11:27:06.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.760+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f5fc4003790 con 0x7f5fc0034850 2024-03-29T11:27:06.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.761+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f5fc4083630 con 0x7f5fc0034850 2024-03-29T11:27:06.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.761+0000 7f5fbdffb640 10 client.4352.objecter ms_dispatch 0x7f5fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:06.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.761+0000 7f5fbdffb640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:06.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.761+0000 7f5fbdffb640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:06.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.761+0000 7f5fbdffb640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:27:06.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.761+0000 7f5fcd682640 10 monclient: get_auth_request con 0x7f5fac06e020 auth_method 0 2024-03-29T11:27:06.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.764+0000 7f5fbdffb640 10 client.4352.objecter ms_handle_connect 0x7f5fac06e020 2024-03-29T11:27:06.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.768+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fc404ed40 con 0x7f5fc0034850 2024-03-29T11:27:06.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.768+0000 7f5fbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:06.768+0000 7f5fbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:07.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.104+0000 7f5fce684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.105+0000 7f5fce684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.105+0000 7f5fce684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:07.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.105+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fa8005330 con 0x7f5fc0034850 2024-03-29T11:27:07.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.107+0000 7f5fbdffb640 1 -- 172.21.15.64:0/2878678966 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f5fc4050e40 con 0x7f5fc0034850 2024-03-29T11:27:07.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.107+0000 7f5fbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.107+0000 7f5fbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:07.096 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:07.096 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:07.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 10 librados: watch_flush enter 2024-03-29T11:27:07.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 10 librados: watch_flush exit 2024-03-29T11:27:07.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:27:07.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:27:07.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 >> v1:172.21.15.136:6816/2312274743 conn(0x7f5fac06e020 legacy=0x7f5fac0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 10 monclient: shutdown 2024-03-29T11:27:07.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:07.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.112+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 >> v1:172.21.15.64:6790/0 conn(0x7f5fc0034850 legacy=0x7f5fc0174d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.117+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 shutdown_connections 2024-03-29T11:27:07.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.117+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 >> 172.21.15.64:0/2878678966 conn(0x7f5fc00058f0 msgr2=0x7f5fc00043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:07.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.117+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 shutdown_connections 2024-03-29T11:27:07.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.117+0000 7f5fce684640 1 -- 172.21.15.64:0/2878678966 wait complete. 2024-03-29T11:27:07.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:07.118+0000 7f5fce684640 1 librados: shutdown 2024-03-29T11:27:07.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:13.931 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.098+0000 7fda2bcb3640 10 monclient: get_monmap_and_config 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.098+0000 7fda2bcb3640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.098+0000 7fda2bcb3640 10 monclient: monmap: 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:14.099308+0000 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:14.099308+0000 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:14.086 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:14.087 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:14.087 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.099+0000 7fda2bcb3640 1 Processor -- start 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.099+0000 7fda2bcb3640 1 -- start start 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.099+0000 7fda2bcb3640 10 monclient: init 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient(hunting): picked mon.noname-a con 0x7fda241457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient(hunting): picked mon.noname-c con 0x7fda2413f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient(hunting): picked mon.noname-b con 0x7fda240a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:14.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda240aaef0 con 0x7fda241457f0 2024-03-29T11:27:14.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda24145bd0 con 0x7fda2413f650 2024-03-29T11:27:14.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda24145eb0 con 0x7fda240a9d70 2024-03-29T11:27:14.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda2bcb3640 10 monclient(hunting): authenticate will time out at 1370.370333s 2024-03-29T11:27:14.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda29a4f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fda241457f0 0x7fda2413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:44746/0 (socket says 172.21.15.64:44746) 2024-03-29T11:27:14.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.100+0000 7fda29a4f640 1 -- 172.21.15.64:0/3729498092 learned_addr learned my addr 172.21.15.64:0/3729498092 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959150830 0 0) 0x7fda24145eb0 con 0x7fda240a9d70 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda10003520 con 0x7fda240a9d70 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200547500 0 0) 0x7fda10003520 con 0x7fda240a9d70 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 >> v1:172.21.15.136:6789/0 conn(0x7fda2413f650 legacy=0x7fda240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:14.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 >> v1:172.21.15.64:6789/0 conn(0x7fda241457f0 legacy=0x7fda2413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 monclient: found mon.noname-b 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24146340 con 0x7fda240a9d70 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fda18001690 con 0x7fda240a9d70 2024-03-29T11:27:14.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: dump: 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:14.093 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda2bcb3640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda2bcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda18001860 con 0x7fda240a9d70 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 1 -- 172.21.15.64:0/3729498092 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fda18001cb0 con 0x7fda240a9d70 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.101+0000 7fda28a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda28a4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda28a4d640 10 monclient: dump: 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.094 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda2bcb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:14.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda2bcb3640 4 set_mon_vals no callback set 2024-03-29T11:27:14.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda2bcb3640 10 monclient: shutdown 2024-03-29T11:27:14.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda2bcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.102+0000 7fda2bcb3640 1 -- 172.21.15.64:0/3729498092 >> v1:172.21.15.64:6790/0 conn(0x7fda240a9d70 legacy=0x7fda2414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.103+0000 7fda2bcb3640 1 -- 172.21.15.64:0/3729498092 shutdown_connections 2024-03-29T11:27:14.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.103+0000 7fda2bcb3640 1 -- 172.21.15.64:0/3729498092 >> 172.21.15.64:0/3729498092 conn(0x7fda24066a00 msgr2=0x7fda24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.103+0000 7fda2bcb3640 1 -- 172.21.15.64:0/3729498092 shutdown_connections 2024-03-29T11:27:14.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.103+0000 7fda2bcb3640 1 -- 172.21.15.64:0/3729498092 wait complete. 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.104+0000 7fda2bcb3640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.104+0000 7fda2bcb3640 10 monclient: monmap: 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:14.106 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.104+0000 7fda2bcb3640 1 librados: starting msgr at 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.104+0000 7fda2bcb3640 1 librados: starting objecter 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.104+0000 7fda2bcb3640 1 Processor -- start 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.105+0000 7fda2bcb3640 1 -- start start 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.105+0000 7fda2bcb3640 1 librados: setting wanted keys 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.105+0000 7fda2bcb3640 1 librados: calling monclient init 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.105+0000 7fda2bcb3640 10 monclient: init 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.105+0000 7fda2bcb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.105+0000 7fda2bcb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 10 monclient(hunting): picked mon.noname-a con 0x7fda240a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 10 monclient(hunting): picked mon.noname-b con 0x7fda2413f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:14.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 10 monclient(hunting): picked mon.noname-c con 0x7fda241457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2406f560 con 0x7fda240a9d70 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda241db980 con 0x7fda2413f650 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda24138d50 con 0x7fda241457f0 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2bcb3640 10 monclient(hunting): authenticate will time out at 1370.376217s 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2924e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fda2413f650 0x7fda2406e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:40354/0 (socket says 172.21.15.64:40354) 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda2924e640 1 -- 172.21.15.64:0/2452822360 learned_addr learned my addr 172.21.15.64:0/2452822360 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 client.?.objecter ms_handle_connect 0x7fda2413f650 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 client.?.objecter ms_handle_connect 0x7fda240a9d70 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425679694 0 0) 0x7fda241db980 con 0x7fda2413f650 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.106+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda08004eb0 con 0x7fda2413f650 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903367945 0 0) 0x7fda2406f560 con 0x7fda240a9d70 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda241db980 con 0x7fda240a9d70 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 10 client.?.objecter ms_handle_connect 0x7fda241457f0 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3501602539 0 0) 0x7fda241db980 con 0x7fda240a9d70 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.107+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda2406f560 con 0x7fda240a9d70 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fda1c001b80 con 0x7fda240a9d70 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): dump: 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:14.110 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136862380 0 0) 0x7fda08004eb0 con 0x7fda2413f650 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda241db980 con 0x7fda2413f650 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662389495 0 0) 0x7fda24138d50 con 0x7fda241457f0 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda08004eb0 con 0x7fda241457f0 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fda200018e0 con 0x7fda2413f650 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:14.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient(hunting): dump: 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2653201858 0 0) 0x7fda2406f560 con 0x7fda240a9d70 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 >> v1:172.21.15.64:6790/0 conn(0x7fda241457f0 legacy=0x7fda24137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 >> v1:172.21.15.136:6789/0 conn(0x7fda2413f650 legacy=0x7fda2406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 1 monclient: found mon.a 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.108+0000 7fda167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24138ee0 con 0x7fda240a9d70 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 10 monclient: _renew_subs 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda24137f90 con 0x7fda240a9d70 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 10 monclient: _renew_subs 2024-03-29T11:27:14.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda241dc080 con 0x7fda240a9d70 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda2bcb3640 1 librados: init done 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda1c002070 con 0x7fda240a9d70 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fda1c004b60 con 0x7fda240a9d70 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.109+0000 7fda167fc640 10 monclient: dump: 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.114 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.110+0000 7fda17fff640 4 set_mon_vals no callback set 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.110+0000 7fda2bcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.110+0000 7fda2bcb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.110+0000 7fda2bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.110+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda24075f10 con 0x7fda240a9d70 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.112+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fda1c004cc0 con 0x7fda240a9d70 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fda1c084ac0 con 0x7fda240a9d70 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda167fc640 10 client.4338.objecter ms_dispatch 0x7fda24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda167fc640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:14.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda167fc640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:14.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda167fc640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:27:14.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda2924e640 10 monclient: get_auth_request con 0x7fda0806de20 auth_method 0 2024-03-29T11:27:14.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.113+0000 7fda167fc640 10 client.4338.objecter ms_handle_connect 0x7fda0806de20 2024-03-29T11:27:14.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.121+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda1c0501d0 con 0x7fda240a9d70 2024-03-29T11:27:14.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.121+0000 7fda167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.121+0000 7fda167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.514+0000 7fda2bcb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.514+0000 7fda2bcb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.514+0000 7fda2bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.514+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda241dbb10 con 0x7fda240a9d70 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.516+0000 7fda167fc640 1 -- 172.21.15.64:0/2452822360 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fda1c0522d0 con 0x7fda240a9d70 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.516+0000 7fda167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.516+0000 7fda167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:14.504 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:14.504 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 10 librados: watch_flush enter 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 10 librados: watch_flush exit 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 >> v1:172.21.15.136:6816/2312274743 conn(0x7fda0806de20 legacy=0x7fda080702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 10 monclient: shutdown 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.519+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 >> v1:172.21.15.64:6789/0 conn(0x7fda240a9d70 legacy=0x7fda2413b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.520+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 shutdown_connections 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.520+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 >> 172.21.15.64:0/2452822360 conn(0x7fda24066a00 msgr2=0x7fda240b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.520+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 shutdown_connections 2024-03-29T11:27:14.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.520+0000 7fda2bcb3640 1 -- 172.21.15.64:0/2452822360 wait complete. 2024-03-29T11:27:14.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:14.520+0000 7fda2bcb3640 1 librados: shutdown 2024-03-29T11:27:14.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:21.329 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:21.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.487+0000 7fdc82f46640 10 monclient: get_monmap_and_config 2024-03-29T11:27:21.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.487+0000 7fdc82f46640 10 monclient: build_initial_monmap 2024-03-29T11:27:21.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.487+0000 7fdc82f46640 10 monclient: monmap: 2024-03-29T11:27:21.476 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:21.476 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:21.476 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:21.488711+0000 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:21.488711+0000 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.488+0000 7fdc82f46640 1 Processor -- start 2024-03-29T11:27:21.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.488+0000 7fdc82f46640 1 -- start start 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.488+0000 7fdc82f46640 10 monclient: init 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c0b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c071eb0 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c143d10 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c06ef80 con 0x7fdc7c143d10 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c06f260 con 0x7fdc7c071eb0 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0b40c0 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:21.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.489+0000 7fdc82f46640 10 monclient(hunting): authenticate will time out at 1377.759786s 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.493+0000 7fdc80ce2640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdc7c0b1f60 0x7fdc7c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34337/0 (socket says 172.21.15.64:34337) 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.493+0000 7fdc80ce2640 1 -- 172.21.15.64:0/2809390797 learned_addr learned my addr 172.21.15.64:0/2809390797 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041898045 0 0) 0x7fdc7c0b40c0 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc64003520 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776539984 0 0) 0x7fdc7c06f260 con 0x7fdc7c071eb0 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc7c0b40c0 con 0x7fdc7c071eb0 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 492051987 0 0) 0x7fdc64003520 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 >> v1:172.21.15.136:6789/0 conn(0x7fdc7c071eb0 legacy=0x7fdc7c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.495+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 >> v1:172.21.15.64:6789/0 conn(0x7fdc7c143d10 legacy=0x7fdc7c146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c155d70 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdc74001a10 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc82f46640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc82f46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc7b7fe640 10 monclient: dump: 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.497+0000 7fdc82f46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc74001c30 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc7b7fe640 1 -- 172.21.15.64:0/2809390797 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdc74001fc0 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc7b7fe640 10 monclient: dump: 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc82f46640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.498+0000 7fdc82f46640 4 set_mon_vals no callback set 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 10 monclient: shutdown 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 1 -- 172.21.15.64:0/2809390797 >> v1:172.21.15.64:6790/0 conn(0x7fdc7c0b1f60 legacy=0x7fdc7c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 1 -- 172.21.15.64:0/2809390797 shutdown_connections 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 1 -- 172.21.15.64:0/2809390797 >> 172.21.15.64:0/2809390797 conn(0x7fdc7c066a00 msgr2=0x7fdc7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 1 -- 172.21.15.64:0/2809390797 shutdown_connections 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.499+0000 7fdc82f46640 1 -- 172.21.15.64:0/2809390797 wait complete. 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.500+0000 7fdc82f46640 10 monclient: build_initial_monmap 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.500+0000 7fdc82f46640 10 monclient: monmap: 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.500+0000 7fdc82f46640 1 librados: starting msgr at 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.500+0000 7fdc82f46640 1 librados: starting objecter 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.500+0000 7fdc82f46640 1 Processor -- start 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.501+0000 7fdc82f46640 1 -- start start 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.501+0000 7fdc82f46640 1 librados: setting wanted keys 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.501+0000 7fdc82f46640 1 librados: calling monclient init 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.501+0000 7fdc82f46640 10 monclient: init 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.501+0000 7fdc82f46640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c0b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c143d10 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c147c70 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c148210 con 0x7fdc7c143d10 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c14ade0 con 0x7fdc7c147c70 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c14b0c0 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.502+0000 7fdc82f46640 10 monclient(hunting): authenticate will time out at 1377.771949s 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.508+0000 7fdc7bfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fdc7c143d10 0x7fdc7c13a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45613/0 (socket says 172.21.15.64:45613) 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.508+0000 7fdc80ce2640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdc7c0b1f60 0x7fdc7c148d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34415/0 (socket says 172.21.15.64:34415) 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.508+0000 7fdc80ce2640 1 -- 172.21.15.64:0/2868298260 learned_addr learned my addr 172.21.15.64:0/2868298260 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:21.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.509+0000 7fdc797fa640 10 client.?.objecter ms_handle_connect 0x7fdc7c143d10 2024-03-29T11:27:21.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.509+0000 7fdc797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 client.?.objecter ms_handle_connect 0x7fdc7c0b1f60 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301053995 0 0) 0x7fdc7c148210 con 0x7fdc7c143d10 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc68004eb0 con 0x7fdc7c143d10 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598016679 0 0) 0x7fdc68004eb0 con 0x7fdc7c143d10 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc68005300 con 0x7fdc7c143d10 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 client.?.objecter ms_handle_connect 0x7fdc7c147c70 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdc70001930 con 0x7fdc7c143d10 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:21.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): dump: 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:21.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.511+0000 7fdc797fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:21.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.512+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514316762 0 0) 0x7fdc7c14b0c0 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.512+0000 7fdc797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.512+0000 7fdc797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.512+0000 7fdc797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.512+0000 7fdc797fa640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc68005850 con 0x7fdc7c0b1f60 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1410478168 0 0) 0x7fdc68005300 con 0x7fdc7c143d10 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 >> v1:172.21.15.64:6790/0 conn(0x7fdc7c0b1f60 legacy=0x7fdc7c148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 >> v1:172.21.15.136:6789/0 conn(0x7fdc7c147c70 legacy=0x7fdc7c13b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 1 monclient: found mon.a 2024-03-29T11:27:21.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c14b3a0 con 0x7fdc7c143d10 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 10 monclient: _renew_subs 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c13b7d0 con 0x7fdc7c143d10 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 10 monclient: _renew_subs 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c070ac0 con 0x7fdc7c143d10 2024-03-29T11:27:21.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.513+0000 7fdc82f46640 1 librados: init done 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.514+0000 7fdc82f46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.515+0000 7fdc82f46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.515+0000 7fdc82f46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.515+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c13b9c0 con 0x7fdc7c143d10 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.516+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc700035f0 con 0x7fdc7c143d10 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdc70001930 con 0x7fdc7c143d10 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 10 monclient: dump: 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:21.507 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.508 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:21.508 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:21.508 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:21.508 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:21.508 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:21.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fdc700038a0 con 0x7fdc7c143d10 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fdc700847f0 con 0x7fdc7c143d10 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc80ce2640 10 monclient: get_auth_request con 0x7fdc6806e220 auth_method 0 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.517+0000 7fdc7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.518+0000 7fdc797fa640 10 client.4341.objecter ms_dispatch 0x7fdc7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.518+0000 7fdc797fa640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.518+0000 7fdc797fa640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.518+0000 7fdc797fa640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:27:21.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.519+0000 7fdc797fa640 10 client.4341.objecter ms_handle_connect 0x7fdc6806e220 2024-03-29T11:27:21.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.523+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc70050150 con 0x7fdc7c143d10 2024-03-29T11:27:21.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.523+0000 7fdc797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.523+0000 7fdc797fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:21.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.859+0000 7fdc82f46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.859+0000 7fdc82f46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.859+0000 7fdc82f46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:21.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.859+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc7c148680 con 0x7fdc7c143d10 2024-03-29T11:27:21.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.860+0000 7fdc797fa640 1 -- 172.21.15.64:0/2868298260 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fdc7c148680 con 0x7fdc7c143d10 2024-03-29T11:27:21.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.860+0000 7fdc797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.860+0000 7fdc797fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:21.848 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:21.848 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:21.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.863+0000 7fdc82f46640 10 librados: watch_flush enter 2024-03-29T11:27:21.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.863+0000 7fdc82f46640 10 librados: watch_flush exit 2024-03-29T11:27:21.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.863+0000 7fdc82f46640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:27:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.863+0000 7fdc82f46640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:27:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.864+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 >> v1:172.21.15.136:6816/2312274743 conn(0x7fdc6806e220 legacy=0x7fdc680706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.864+0000 7fdc82f46640 10 monclient: shutdown 2024-03-29T11:27:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.864+0000 7fdc82f46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.864+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 >> v1:172.21.15.64:6789/0 conn(0x7fdc7c143d10 legacy=0x7fdc7c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.866+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 shutdown_connections 2024-03-29T11:27:21.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.866+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 >> 172.21.15.64:0/2868298260 conn(0x7fdc7c066a00 msgr2=0x7fdc7c142a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.870+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 shutdown_connections 2024-03-29T11:27:21.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.870+0000 7fdc82f46640 1 -- 172.21.15.64:0/2868298260 wait complete. 2024-03-29T11:27:21.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:21.870+0000 7fdc82f46640 1 librados: shutdown 2024-03-29T11:27:21.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:28.686 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:28.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.859+0000 7f49c64ae640 10 monclient: get_monmap_and_config 2024-03-29T11:27:28.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.859+0000 7f49c64ae640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.859+0000 7f49c64ae640 10 monclient: monmap: 2024-03-29T11:27:28.847 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:28.847 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.847 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:28.860172+0000 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:28.860172+0000 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.860+0000 7f49c64ae640 1 Processor -- start 2024-03-29T11:27:28.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.860+0000 7f49c64ae640 1 -- start start 2024-03-29T11:27:28.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.860+0000 7f49c64ae640 10 monclient: init 2024-03-29T11:27:28.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:28.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient(hunting): picked mon.noname-a con 0x7f49c00b0500 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:28.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient(hunting): picked mon.noname-c con 0x7f49c00b08e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient(hunting): picked mon.noname-b con 0x7f49c00aeca0 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c0140620 con 0x7f49c00b0500 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c013daf0 con 0x7f49c00b08e0 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00ae180 con 0x7f49c00aeca0 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49c64ae640 10 monclient(hunting): authenticate will time out at 1385.131412s 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49bffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f49c00b0500 0x7f49c013ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:53342/0 (socket says 172.21.15.64:53342) 2024-03-29T11:27:28.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.861+0000 7f49bffff640 1 -- 172.21.15.64:0/4227528227 learned_addr learned my addr 172.21.15.64:0/4227528227 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:28.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936717320 0 0) 0x7f49c00ae180 con 0x7f49c00aeca0 2024-03-29T11:27:28.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a8003520 con 0x7f49c00aeca0 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443194578 0 0) 0x7f49c013daf0 con 0x7f49c00b08e0 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c00ae180 con 0x7f49c00b08e0 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990883361 0 0) 0x7f49a8003520 con 0x7f49c00aeca0 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 >> v1:172.21.15.136:6789/0 conn(0x7f49c00b08e0 legacy=0x7f49c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 >> v1:172.21.15.64:6789/0 conn(0x7f49c00b0500 legacy=0x7f49c013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00accf0 con 0x7f49c00aeca0 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f49b8001990 con 0x7f49c00aeca0 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.862+0000 7f49beffd640 10 monclient: dump: 2024-03-29T11:27:28.853 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:28.854 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49c64ae640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49c64ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b8001af0 con 0x7f49c00aeca0 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49beffd640 1 -- 172.21.15.64:0/4227528227 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f49b8001f40 con 0x7f49c00aeca0 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49beffd640 10 monclient: dump: 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.855 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49c64ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.863+0000 7f49c64ae640 4 set_mon_vals no callback set 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 10 monclient: shutdown 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 1 -- 172.21.15.64:0/4227528227 >> v1:172.21.15.64:6790/0 conn(0x7f49c00aeca0 legacy=0x7f49c014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 1 -- 172.21.15.64:0/4227528227 shutdown_connections 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 1 -- 172.21.15.64:0/4227528227 >> 172.21.15.64:0/4227528227 conn(0x7f49c006b660 msgr2=0x7f49c006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 1 -- 172.21.15.64:0/4227528227 shutdown_connections 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.864+0000 7f49c64ae640 1 -- 172.21.15.64:0/4227528227 wait complete. 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.865+0000 7f49c64ae640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.865+0000 7f49c64ae640 10 monclient: monmap: 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 1 librados: starting msgr at 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 1 librados: starting objecter 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 1 Processor -- start 2024-03-29T11:27:28.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 1 -- start start 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 1 librados: setting wanted keys 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 1 librados: calling monclient init 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.866+0000 7f49c64ae640 10 monclient: init 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient(hunting): picked mon.noname-b con 0x7f49c00aeca0 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient(hunting): picked mon.noname-a con 0x7f49c00b0500 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient(hunting): picked mon.noname-c con 0x7f49c011e530 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c01c6fb0 con 0x7f49c00b0500 2024-03-29T11:27:28.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c011e010 con 0x7f49c00aeca0 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c011f0d0 con 0x7f49c011e530 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49c64ae640 10 monclient(hunting): authenticate will time out at 1385.137487s 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49bf7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f49c00b0500 0x7f49c01df4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:53456/0 (socket says 172.21.15.64:53456) 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.867+0000 7f49bf7fe640 1 -- 172.21.15.64:0/4259896108 learned_addr learned my addr 172.21.15.64:0/4259896108 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 client.?.objecter ms_handle_connect 0x7f49c00b0500 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 client.?.objecter ms_handle_connect 0x7f49c011e530 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 client.?.objecter ms_handle_connect 0x7f49c00aeca0 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067006116 0 0) 0x7f49c01c6fb0 con 0x7f49c00b0500 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac004eb0 con 0x7f49c00b0500 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691232408 0 0) 0x7f49c011e010 con 0x7f49c00aeca0 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c01c6fb0 con 0x7f49c00aeca0 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952304795 0 0) 0x7f49c011f0d0 con 0x7f49c011e530 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.868+0000 7f49bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c011e010 con 0x7f49c011e530 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1130293814 0 0) 0x7f49ac004eb0 con 0x7f49c00b0500 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49c011f0d0 con 0x7f49c00b0500 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f49b0001940 con 0x7f49c00b0500 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): dump: 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.870 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126357839 0 0) 0x7f49c01c6fb0 con 0x7f49c00aeca0 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49ac004eb0 con 0x7f49c00aeca0 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f49b40018b0 con 0x7f49c00aeca0 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): dump: 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.871 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1752443664 0 0) 0x7f49c011e010 con 0x7f49c011e530 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49c01c6fb0 con 0x7f49c011e530 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f49b8001c60 con 0x7f49c011e530 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 10 monclient(hunting): dump: 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.872 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1576847132 0 0) 0x7f49c011f0d0 con 0x7f49c00b0500 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.869+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 >> v1:172.21.15.64:6790/0 conn(0x7f49c011e530 legacy=0x7f49c011e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 >> v1:172.21.15.136:6789/0 conn(0x7f49c00aeca0 legacy=0x7f49c01dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 1 monclient: found mon.a 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:28.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c01ea0e0 con 0x7f49c00b0500 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 10 monclient: _renew_subs 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c01e70b0 con 0x7f49c00b0500 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 10 monclient: _renew_subs 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c01e7540 con 0x7f49c00b0500 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49c64ae640 1 librados: init done 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b0001aa0 con 0x7f49c00b0500 2024-03-29T11:27:28.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bdffb640 4 set_mon_vals no callback set 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f49b00038c0 con 0x7f49c00b0500 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.870+0000 7f49bcff9640 10 monclient: dump: 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:28.875 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.872+0000 7f49c64ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.872+0000 7f49c64ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.872+0000 7f49c64ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.872+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a0005080 con 0x7f49c00b0500 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.873+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f49b0003a20 con 0x7f49c00b0500 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.874+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f49b0084a00 con 0x7f49c00b0500 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.874+0000 7f49bcff9640 10 client.4347.objecter ms_dispatch 0x7f49c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.874+0000 7f49bcff9640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.874+0000 7f49bcff9640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.874+0000 7f49bcff9640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.874+0000 7f49bffff640 10 monclient: get_auth_request con 0x7f49ac06e420 auth_method 0 2024-03-29T11:27:28.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.875+0000 7f49bcff9640 10 client.4347.objecter ms_handle_connect 0x7f49ac06e420 2024-03-29T11:27:28.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.883+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49b00502e0 con 0x7f49c00b0500 2024-03-29T11:27:28.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.883+0000 7f49bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:28.883+0000 7f49bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:29.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.271+0000 7f49c64ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.271+0000 7f49c64ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.271+0000 7f49c64ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:29.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.271+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49a0005330 con 0x7f49c00b0500 2024-03-29T11:27:29.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.272+0000 7f49bcff9640 1 -- 172.21.15.64:0/4259896108 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f49b0052210 con 0x7f49c00b0500 2024-03-29T11:27:29.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.272+0000 7f49bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.272+0000 7f49bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:29.260 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:29.261 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:29.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 10 librados: watch_flush enter 2024-03-29T11:27:29.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 10 librados: watch_flush exit 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 >> v1:172.21.15.136:6816/2312274743 conn(0x7f49ac06e420 legacy=0x7f49ac0708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 10 monclient: shutdown 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 >> v1:172.21.15.64:6789/0 conn(0x7f49c00b0500 legacy=0x7f49c01df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 shutdown_connections 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 >> 172.21.15.64:0/4259896108 conn(0x7f49c006b660 msgr2=0x7f49c00a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 shutdown_connections 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 -- 172.21.15.64:0/4259896108 wait complete. 2024-03-29T11:27:29.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:29.276+0000 7f49c64ae640 1 librados: shutdown 2024-03-29T11:27:29.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:36.088 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:36.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.250+0000 7fc486206640 10 monclient: get_monmap_and_config 2024-03-29T11:27:36.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.250+0000 7fc486206640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.250+0000 7fc486206640 10 monclient: monmap: 2024-03-29T11:27:36.238 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:36.251045+0000 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:36.251045+0000 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:36.239 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.251+0000 7fc486206640 1 Processor -- start 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.251+0000 7fc486206640 1 -- start start 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.251+0000 7fc486206640 10 monclient: init 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient(hunting): picked mon.noname-a con 0x7fc4800b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient(hunting): picked mon.noname-b con 0x7fc48014a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient(hunting): picked mon.noname-c con 0x7fc4800ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800aba70 con 0x7fc4800b0770 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800aecb0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800af150 con 0x7fc48014a700 2024-03-29T11:27:36.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.252+0000 7fc486206640 10 monclient(hunting): authenticate will time out at 1392.521977s 2024-03-29T11:27:36.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.253+0000 7fc47ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc4800ac5a0 0x7fc480151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:37686/0 (socket says 172.21.15.64:37686) 2024-03-29T11:27:36.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.253+0000 7fc47ffff640 1 -- 172.21.15.64:0/1261435171 learned_addr learned my addr 172.21.15.64:0/1261435171 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:36.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.254+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181286879 0 0) 0x7fc4800aecb0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.254+0000 7fc47e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.254+0000 7fc47e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.254+0000 7fc47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.254+0000 7fc47e7fc640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:27:36.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.254+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc468003520 con 0x7fc4800ac5a0 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204552395 0 0) 0x7fc468003520 con 0x7fc4800ac5a0 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 >> v1:172.21.15.64:6790/0 conn(0x7fc48014a700 legacy=0x7fc4800abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 >> v1:172.21.15.64:6789/0 conn(0x7fc4800b0770 legacy=0x7fc480149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:27:36.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4800ac9d0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc47e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc486206640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.255+0000 7fc486206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.256+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc478001a30 con 0x7fc4800ac5a0 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.256+0000 7fc47e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.256+0000 7fc47e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.256+0000 7fc47e7fc640 10 monclient: dump: 2024-03-29T11:27:36.246 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.256+0000 7fc486206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:36.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.257+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc478001cc0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc47e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.249 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc486206640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc486206640 4 set_mon_vals no callback set 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc47e7fc640 1 -- 172.21.15.64:0/1261435171 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc478002050 con 0x7fc4800ac5a0 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc47e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc47e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc47e7fc640 10 monclient: dump: 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.250 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc486206640 10 monclient: shutdown 2024-03-29T11:27:36.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc486206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.258+0000 7fc486206640 1 -- 172.21.15.64:0/1261435171 >> v1:172.21.15.136:6789/0 conn(0x7fc4800ac5a0 legacy=0x7fc480151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.259+0000 7fc486206640 1 -- 172.21.15.64:0/1261435171 shutdown_connections 2024-03-29T11:27:36.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.259+0000 7fc486206640 1 -- 172.21.15.64:0/1261435171 >> 172.21.15.64:0/1261435171 conn(0x7fc48006b660 msgr2=0x7fc48006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.260+0000 7fc486206640 1 -- 172.21.15.64:0/1261435171 shutdown_connections 2024-03-29T11:27:36.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.261+0000 7fc486206640 1 -- 172.21.15.64:0/1261435171 wait complete. 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.262+0000 7fc486206640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.262+0000 7fc486206640 10 monclient: monmap: 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:36.253 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.262+0000 7fc486206640 1 librados: starting msgr at 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.262+0000 7fc486206640 1 librados: starting objecter 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.262+0000 7fc486206640 1 Processor -- start 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.263+0000 7fc486206640 1 -- start start 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.263+0000 7fc486206640 1 librados: setting wanted keys 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.263+0000 7fc486206640 1 librados: calling monclient init 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.263+0000 7fc486206640 10 monclient: init 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient(hunting): picked mon.noname-c con 0x7fc4800ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient(hunting): picked mon.noname-a con 0x7fc4800b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:36.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient(hunting): picked mon.noname-b con 0x7fc48014a700 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:36.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4801272a0 con 0x7fc4800b0770 2024-03-29T11:27:36.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc480128400 con 0x7fc48014a700 2024-03-29T11:27:36.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4801db550 con 0x7fc4800ac5a0 2024-03-29T11:27:36.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.264+0000 7fc486206640 10 monclient(hunting): authenticate will time out at 1392.534519s 2024-03-29T11:27:36.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.267+0000 7fc47f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc4800ac5a0 0x7fc480126b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53379/0 (socket says 172.21.15.64:53379) 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.267+0000 7fc47f7fe640 1 -- 172.21.15.64:0/120251603 learned_addr learned my addr 172.21.15.64:0/120251603 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.267+0000 7fc467fff640 10 client.?.objecter ms_handle_connect 0x7fc4800ac5a0 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.267+0000 7fc467fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.267+0000 7fc467fff640 10 client.?.objecter ms_handle_connect 0x7fc48014a700 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.267+0000 7fc467fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456179169 0 0) 0x7fc480128400 con 0x7fc48014a700 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:27:36.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c004eb0 con 0x7fc48014a700 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452980392 0 0) 0x7fc4801db550 con 0x7fc4800ac5a0 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc480128400 con 0x7fc4800ac5a0 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752966975 0 0) 0x7fc46c004eb0 con 0x7fc48014a700 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4801db550 con 0x7fc48014a700 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc4780018f0 con 0x7fc48014a700 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): dump: 2024-03-29T11:27:36.268 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2656270427 0 0) 0x7fc480128400 con 0x7fc4800ac5a0 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc46c004eb0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc470001910 con 0x7fc4800ac5a0 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 10 monclient(hunting): dump: 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3351382974 0 0) 0x7fc46c004eb0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 >> v1:172.21.15.136:6789/0 conn(0x7fc48014a700 legacy=0x7fc4801450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.268+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 >> v1:172.21.15.64:6789/0 conn(0x7fc4800b0770 legacy=0x7fc4801dba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 1 monclient: found mon.c 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4801239d0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 10 monclient: _renew_subs 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4801dc180 con 0x7fc4800ac5a0 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 10 monclient: _renew_subs 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4801459a0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.269+0000 7fc486206640 1 librados: init done 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc486206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc486206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc486206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc480123be0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4700035f0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc470001910 con 0x7fc4800ac5a0 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc467fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.271+0000 7fc467fff640 10 monclient: dump: 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:36.273 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:36.274 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:36.274 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:36.274 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:36.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.272+0000 7fc47dffb640 4 set_mon_vals no callback set 2024-03-29T11:27:36.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.272+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fc4700038a0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.275+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fc470054500 con 0x7fc4800ac5a0 2024-03-29T11:27:36.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.275+0000 7fc467fff640 10 client.4373.objecter ms_dispatch 0x7fc480000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:36.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.275+0000 7fc467fff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:36.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.275+0000 7fc467fff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:36.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.275+0000 7fc467fff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:27:36.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.275+0000 7fc47effd640 10 monclient: get_auth_request con 0x7fc46c06e090 auth_method 0 2024-03-29T11:27:36.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.276+0000 7fc467fff640 10 client.4373.objecter ms_handle_connect 0x7fc46c06e090 2024-03-29T11:27:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.283+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4700507a0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.283+0000 7fc467fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.283+0000 7fc467fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.624+0000 7fc486206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.624+0000 7fc486206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.624+0000 7fc486206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.624+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc480128220 con 0x7fc4800ac5a0 2024-03-29T11:27:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.626+0000 7fc467fff640 1 -- 172.21.15.64:0/120251603 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fc4700507a0 con 0x7fc4800ac5a0 2024-03-29T11:27:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.626+0000 7fc467fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.626+0000 7fc467fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:36.614 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:36.614 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 10 librados: watch_flush enter 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 10 librados: watch_flush exit 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 >> v1:172.21.15.136:6816/2312274743 conn(0x7fc46c06e090 legacy=0x7fc46c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 10 monclient: shutdown 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.629+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 >> v1:172.21.15.64:6790/0 conn(0x7fc4800ac5a0 legacy=0x7fc480126b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.632+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 shutdown_connections 2024-03-29T11:27:36.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.632+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 >> 172.21.15.64:0/120251603 conn(0x7fc48006b660 msgr2=0x7fc4800b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.632+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 shutdown_connections 2024-03-29T11:27:36.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.632+0000 7fc486206640 1 -- 172.21.15.64:0/120251603 wait complete. 2024-03-29T11:27:36.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:36.632+0000 7fc486206640 1 librados: shutdown 2024-03-29T11:27:36.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:43.445 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:43.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.601+0000 7f8c39025640 10 monclient: get_monmap_and_config 2024-03-29T11:27:43.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.601+0000 7f8c39025640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.601+0000 7f8c39025640 10 monclient: monmap: 2024-03-29T11:27:43.592 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:43.592 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:43.592 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:43.603869+0000 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:43.603869+0000 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.602+0000 7f8c39025640 1 Processor -- start 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.602+0000 7f8c39025640 1 -- start start 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.602+0000 7f8c39025640 10 monclient: init 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient(hunting): picked mon.noname-c con 0x7f8c340b4100 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient(hunting): picked mon.noname-a con 0x7f8c3413ee90 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient(hunting): picked mon.noname-b con 0x7f8c340a9ac0 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340af6f0 con 0x7f8c3413ee90 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340aac80 con 0x7f8c340b4100 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340aaf60 con 0x7f8c340a9ac0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.603+0000 7f8c39025640 10 monclient(hunting): authenticate will time out at 1399.874695s 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.607+0000 7f8c32d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8c340b4100 0x7f8c340adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:41068/0 (socket says 172.21.15.64:41068) 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.607+0000 7f8c32d9d640 1 -- 172.21.15.64:0/2501418734 learned_addr learned my addr 172.21.15.64:0/2501418734 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:43.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871946862 0 0) 0x7f8c340aac80 con 0x7f8c340b4100 2024-03-29T11:27:43.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1c003520 con 0x7f8c340b4100 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474866413 0 0) 0x7f8c340af6f0 con 0x7f8c3413ee90 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.608+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c340aac80 con 0x7f8c3413ee90 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2817117343 0 0) 0x7f8c1c003520 con 0x7f8c340b4100 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 >> v1:172.21.15.64:6790/0 conn(0x7f8c340a9ac0 legacy=0x7f8c34142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:43.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 >> v1:172.21.15.64:6789/0 conn(0x7f8c3413ee90 legacy=0x7f8c3413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 1 monclient: found mon.noname-c 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3413e970 con 0x7f8c340b4100 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c2c001a10 con 0x7f8c340b4100 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.610+0000 7f8c31d9b640 10 monclient: dump: 2024-03-29T11:27:43.602 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c2c001b70 con 0x7f8c340b4100 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c31d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c31d9b640 1 -- 172.21.15.64:0/2501418734 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c2c001f80 con 0x7f8c340b4100 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c31d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c31d9b640 10 monclient: dump: 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c39025640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c39025640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:43.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.611+0000 7f8c39025640 4 set_mon_vals no callback set 2024-03-29T11:27:43.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 10 monclient: shutdown 2024-03-29T11:27:43.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 1 -- 172.21.15.64:0/2501418734 >> v1:172.21.15.136:6789/0 conn(0x7f8c340b4100 legacy=0x7f8c340adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 1 -- 172.21.15.64:0/2501418734 shutdown_connections 2024-03-29T11:27:43.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 1 -- 172.21.15.64:0/2501418734 >> 172.21.15.64:0/2501418734 conn(0x7f8c340b35e0 msgr2=0x7f8c340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 1 -- 172.21.15.64:0/2501418734 shutdown_connections 2024-03-29T11:27:43.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.612+0000 7f8c39025640 1 -- 172.21.15.64:0/2501418734 wait complete. 2024-03-29T11:27:43.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.613+0000 7f8c39025640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.613+0000 7f8c39025640 10 monclient: monmap: 2024-03-29T11:27:43.610 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:43.610 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.613+0000 7f8c39025640 1 librados: starting msgr at 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.613+0000 7f8c39025640 1 librados: starting objecter 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 1 Processor -- start 2024-03-29T11:27:43.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 1 -- start start 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 1 librados: setting wanted keys 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 1 librados: calling monclient init 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 10 monclient: init 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.614+0000 7f8c39025640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 10 monclient(hunting): picked mon.noname-a con 0x7f8c340a9ac0 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:43.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 10 monclient(hunting): picked mon.noname-b con 0x7f8c340b4100 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 10 monclient(hunting): picked mon.noname-c con 0x7f8c3413ee90 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c34142590 con 0x7f8c340a9ac0 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c34070660 con 0x7f8c340b4100 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c34070940 con 0x7f8c3413ee90 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c39025640 10 monclient(hunting): authenticate will time out at 1399.888300s 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c32d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8c340a9ac0 0x7f8c3406eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45921/0 (socket says 172.21.15.64:45921) 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.617+0000 7f8c32d9d640 1 -- 172.21.15.64:0/1965227117 learned_addr learned my addr 172.21.15.64:0/1965227117 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.618+0000 7f8c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f8c340a9ac0 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.618+0000 7f8c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.618+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963909218 0 0) 0x7f8c34142590 con 0x7f8c340a9ac0 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c20004eb0 con 0x7f8c340a9ac0 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413189311 0 0) 0x7f8c20004eb0 con 0x7f8c340a9ac0 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c34142590 con 0x7f8c340a9ac0 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c2c001fc0 con 0x7f8c340a9ac0 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.614 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f8c340b4100 2024-03-29T11:27:43.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.619+0000 7f8c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.622+0000 7f8c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f8c3413ee90 2024-03-29T11:27:43.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.622+0000 7f8c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.624+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2978084000 0 0) 0x7f8c34142590 con 0x7f8c340a9ac0 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.624+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 >> v1:172.21.15.64:6790/0 conn(0x7f8c3413ee90 legacy=0x7f8c34083750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.624+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 >> v1:172.21.15.136:6789/0 conn(0x7f8c340b4100 legacy=0x7f8c34080020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 1 monclient: found mon.a 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c34070d60 con 0x7f8c340a9ac0 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 10 monclient: _renew_subs 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:43.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3412bff0 con 0x7f8c340a9ac0 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 10 monclient: _renew_subs 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3413df80 con 0x7f8c340a9ac0 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c39025640 1 librados: init done 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c2c004e50 con 0x7f8c340a9ac0 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c3159a640 4 set_mon_vals no callback set 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.625+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8c2c005220 con 0x7f8c340a9ac0 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.626+0000 7f8c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.626+0000 7f8c1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.626+0000 7f8c1b7fe640 10 monclient: dump: 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:43.621 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:43.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.626+0000 7f8c39025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.627+0000 7f8c39025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.627+0000 7f8c39025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.627+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14005080 con 0x7f8c340a9ac0 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.627+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f8c2c005380 con 0x7f8c340a9ac0 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.628+0000 7f8c3259c640 10 monclient: get_auth_request con 0x7f8c2006dd60 auth_method 0 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.628+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f8c2c085020 con 0x7f8c340a9ac0 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.628+0000 7f8c1b7fe640 10 client.4356.objecter ms_dispatch 0x7f8c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.628+0000 7f8c1b7fe640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.629+0000 7f8c1b7fe640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.629+0000 7f8c1b7fe640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:27:43.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.629+0000 7f8c1b7fe640 10 client.4356.objecter ms_handle_connect 0x7f8c2006dd60 2024-03-29T11:27:43.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.636+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c2c050900 con 0x7f8c340a9ac0 2024-03-29T11:27:43.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.636+0000 7f8c1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.636+0000 7f8c1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:43.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.969+0000 7f8c39025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.969+0000 7f8c39025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.969+0000 7f8c39025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:43.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.969+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c14005330 con 0x7f8c340a9ac0 2024-03-29T11:27:43.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.969+0000 7f8c1b7fe640 1 -- 172.21.15.64:0/1965227117 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f8c2c052830 con 0x7f8c340a9ac0 2024-03-29T11:27:43.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.970+0000 7f8c1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.970+0000 7f8c1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:43.960 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:43.960 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:43.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.974+0000 7f8c39025640 10 librados: watch_flush enter 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.975+0000 7f8c39025640 10 librados: watch_flush exit 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.975+0000 7f8c39025640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.975+0000 7f8c39025640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.976+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 >> v1:172.21.15.136:6816/2312274743 conn(0x7f8c2006dd60 legacy=0x7f8c20070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.976+0000 7f8c39025640 10 monclient: shutdown 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.976+0000 7f8c39025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.976+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 >> v1:172.21.15.64:6789/0 conn(0x7f8c340a9ac0 legacy=0x7f8c3406eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.976+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 shutdown_connections 2024-03-29T11:27:43.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.976+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 >> 172.21.15.64:0/1965227117 conn(0x7f8c340b35e0 msgr2=0x7f8c3412ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.977+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 shutdown_connections 2024-03-29T11:27:43.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.977+0000 7f8c39025640 1 -- 172.21.15.64:0/1965227117 wait complete. 2024-03-29T11:27:43.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:43.977+0000 7f8c39025640 1 librados: shutdown 2024-03-29T11:27:43.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:50.791 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:50.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.955+0000 7fc4c128d640 10 monclient: get_monmap_and_config 2024-03-29T11:27:50.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.955+0000 7fc4c128d640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.955+0000 7fc4c128d640 10 monclient: monmap: 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:50.956924+0000 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:50.956924+0000 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:50.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.957+0000 7fc4c128d640 1 Processor -- start 2024-03-29T11:27:50.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.957+0000 7fc4c128d640 1 -- start start 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.957+0000 7fc4c128d640 10 monclient: init 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4b4034880 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4b402a670 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4b4032800 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b402c850 con 0x7fc4b402a670 2024-03-29T11:27:50.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b402a150 con 0x7fc4b4034880 2024-03-29T11:27:50.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b40c75e0 con 0x7fc4b4032800 2024-03-29T11:27:50.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.958+0000 7fc4c128d640 10 monclient(hunting): authenticate will time out at 1407.228046s 2024-03-29T11:27:50.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.961+0000 7fc4bb7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc4b4032800 0x7fc4b40cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34891/0 (socket says 172.21.15.64:34891) 2024-03-29T11:27:50.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.961+0000 7fc4bb7fe640 1 -- 172.21.15.64:0/3286437014 learned_addr learned my addr 172.21.15.64:0/3286437014 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071510732 0 0) 0x7fc4b40c75e0 con 0x7fc4b4032800 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003520 con 0x7fc4b4032800 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465694273 0 0) 0x7fc4b402c850 con 0x7fc4b402a670 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b40c75e0 con 0x7fc4b402a670 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.965+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515630122 0 0) 0x7fc4a4003520 con 0x7fc4b4032800 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 >> v1:172.21.15.136:6789/0 conn(0x7fc4b4034880 legacy=0x7fc4b40c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 >> v1:172.21.15.64:6789/0 conn(0x7fc4b402a670 legacy=0x7fc4b40320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4033140 con 0x7fc4b4032800 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc4b0001690 con 0x7fc4b4032800 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: dump: 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b00018b0 con 0x7fc4b4032800 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 1 -- 172.21.15.64:0/3286437014 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc4b0001c40 con 0x7fc4b4032800 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.966+0000 7fc4b9ffb640 10 monclient: dump: 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.968+0000 7fc4c128d640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.968+0000 7fc4c128d640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.968+0000 7fc4c128d640 4 set_mon_vals no callback set 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.968+0000 7fc4c128d640 10 monclient: shutdown 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.968+0000 7fc4c128d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.968+0000 7fc4c128d640 1 -- 172.21.15.64:0/3286437014 >> v1:172.21.15.64:6790/0 conn(0x7fc4b4032800 legacy=0x7fc4b40cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.969+0000 7fc4c128d640 1 -- 172.21.15.64:0/3286437014 shutdown_connections 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.969+0000 7fc4c128d640 1 -- 172.21.15.64:0/3286437014 >> 172.21.15.64:0/3286437014 conn(0x7fc4b4005920 msgr2=0x7fc4b4005d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.969+0000 7fc4c128d640 1 -- 172.21.15.64:0/3286437014 shutdown_connections 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.969+0000 7fc4c128d640 1 -- 172.21.15.64:0/3286437014 wait complete. 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.970+0000 7fc4c128d640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.970+0000 7fc4c128d640 10 monclient: monmap: 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 1 librados: starting msgr at 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 1 librados: starting objecter 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 1 Processor -- start 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 1 -- start start 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 1 librados: setting wanted keys 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 1 librados: calling monclient init 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.971+0000 7fc4c128d640 10 monclient: init 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4b402a670 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4b4032800 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4b4034880 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b40bf0d0 con 0x7fc4b4032800 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b4164f90 con 0x7fc4b402a670 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b4164b40 con 0x7fc4b4034880 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.972+0000 7fc4c128d640 10 monclient(hunting): authenticate will time out at 1407.242089s 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.978+0000 7fc4baffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc4b402a670 0x7fc4b4164430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:43525/0 (socket says 172.21.15.64:43525) 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.978+0000 7fc4bb7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc4b4034880 0x7fc4b40bd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34985/0 (socket says 172.21.15.64:34985) 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.978+0000 7fc4bb7fe640 1 -- 172.21.15.64:0/699955724 learned_addr learned my addr 172.21.15.64:0/699955724 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.978+0000 7fc4ba7fc640 1 --1- 172.21.15.64:0/699955724 >> v1:172.21.15.64:6789/0 conn(0x7fc4b4032800 0x7fc4b40bd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:50233/0 (socket says 172.21.15.64:50233) 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.978+0000 7fc4a37fe640 10 client.?.objecter ms_handle_connect 0x7fc4b4034880 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 client.?.objecter ms_handle_connect 0x7fc4b4032800 2024-03-29T11:27:50.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 client.?.objecter ms_handle_connect 0x7fc4b402a670 2024-03-29T11:27:50.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542122529 0 0) 0x7fc4b40bf0d0 con 0x7fc4b4032800 2024-03-29T11:27:50.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:27:50.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.979+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8004eb0 con 0x7fc4b4032800 2024-03-29T11:27:50.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294061140 0 0) 0x7fc4a8004eb0 con 0x7fc4b4032800 2024-03-29T11:27:50.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4b40bf0d0 con 0x7fc4b4032800 2024-03-29T11:27:50.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc4ac001950 con 0x7fc4b4032800 2024-03-29T11:27:50.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:50.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 10 monclient(hunting): dump: 2024-03-29T11:27:50.992 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:50.992 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:50.992 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:50.993 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:50.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:50.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3834913437 0 0) 0x7fc4b40bf0d0 con 0x7fc4b4032800 2024-03-29T11:27:50.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 >> v1:172.21.15.64:6790/0 conn(0x7fc4b4034880 legacy=0x7fc4b40bd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.980+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 >> v1:172.21.15.136:6789/0 conn(0x7fc4b402a670 legacy=0x7fc4b4164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 1 monclient: found mon.a 2024-03-29T11:27:50.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:50.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4164cd0 con 0x7fc4b4032800 2024-03-29T11:27:50.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:27:50.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 10 monclient: _renew_subs 2024-03-29T11:27:50.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:50.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b40d2fb0 con 0x7fc4b4032800 2024-03-29T11:27:50.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:50.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 10 monclient: _renew_subs 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b416c8f0 con 0x7fc4b4032800 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.981+0000 7fc4c128d640 1 librados: init done 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4ac0035f0 con 0x7fc4b4032800 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc4ac003a00 con 0x7fc4b4032800 2024-03-29T11:27:50.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:50.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 10 monclient: dump: 2024-03-29T11:27:50.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:50.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:50.998 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.998 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4b8ff9640 4 set_mon_vals no callback set 2024-03-29T11:27:50.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.984+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fc4ac003c20 con 0x7fc4b4032800 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.985+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fc4ac084aa0 con 0x7fc4b4032800 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.985+0000 7fc4a37fe640 10 client.4362.objecter ms_dispatch 0x7fc4bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.985+0000 7fc4a37fe640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.985+0000 7fc4a37fe640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.985+0000 7fc4a37fe640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.987+0000 7fc4baffd640 10 monclient: get_auth_request con 0x7fc4a806dd30 auth_method 0 2024-03-29T11:27:51.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.987+0000 7fc4c128d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:51.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.987+0000 7fc4c128d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:51.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.987+0000 7fc4c128d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:51.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.987+0000 7fc4a37fe640 10 client.4362.objecter ms_handle_connect 0x7fc4a806dd30 2024-03-29T11:27:51.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.987+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49c005080 con 0x7fc4b4032800 2024-03-29T11:27:51.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.995+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4ac050500 con 0x7fc4b4032800 2024-03-29T11:27:51.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.996+0000 7fc4a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:51.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:50.996+0000 7fc4a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:51.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.328+0000 7fc4c128d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.328+0000 7fc4c128d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.328+0000 7fc4c128d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:27:51.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.328+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc49c005330 con 0x7fc4b4032800 2024-03-29T11:27:51.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.332+0000 7fc4a37fe640 1 -- 172.21.15.64:0/699955724 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fc4ac054930 con 0x7fc4b4032800 2024-03-29T11:27:51.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.332+0000 7fc4a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.332+0000 7fc4a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:51.321 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:51.321 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:51.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.335+0000 7fc4c128d640 10 librados: watch_flush enter 2024-03-29T11:27:51.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 10 librados: watch_flush exit 2024-03-29T11:27:51.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:27:51.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 >> v1:172.21.15.136:6816/2312274743 conn(0x7fc4a806dd30 legacy=0x7fc4a80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 10 monclient: shutdown 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 >> v1:172.21.15.64:6789/0 conn(0x7fc4b4032800 legacy=0x7fc4b40bd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 shutdown_connections 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.338+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 >> 172.21.15.64:0/699955724 conn(0x7fc4b4005920 msgr2=0x7fc4b40c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:51.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.339+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 shutdown_connections 2024-03-29T11:27:51.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.339+0000 7fc4c128d640 1 -- 172.21.15.64:0/699955724 wait complete. 2024-03-29T11:27:51.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:51.339+0000 7fc4c128d640 1 librados: shutdown 2024-03-29T11:27:51.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:58.151 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:58.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.316+0000 7f9def37b640 10 monclient: get_monmap_and_config 2024-03-29T11:27:58.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.316+0000 7f9def37b640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.316+0000 7f9def37b640 10 monclient: monmap: 2024-03-29T11:27:58.306 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:27:58.317848+0000 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:27:58.317848+0000 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:58.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.317+0000 7f9def37b640 1 Processor -- start 2024-03-29T11:27:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.317+0000 7f9def37b640 1 -- start start 2024-03-29T11:27:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.317+0000 7f9def37b640 10 monclient: init 2024-03-29T11:27:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.318+0000 7f9def37b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.318+0000 7f9def37b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.318+0000 7f9def37b640 10 monclient(hunting): picked mon.noname-a con 0x7f9de80a9ac0 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.318+0000 7f9def37b640 10 monclient(hunting): picked mon.noname-b con 0x7f9de813ee90 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.318+0000 7f9def37b640 10 monclient(hunting): picked mon.noname-c con 0x7f9de80af900 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.319+0000 7f9def37b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80afd10 con 0x7f9de80a9ac0 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.319+0000 7f9def37b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80af6f0 con 0x7f9de80af900 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.319+0000 7f9def37b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80aac80 con 0x7f9de813ee90 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.319+0000 7f9def37b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.319+0000 7f9def37b640 10 monclient(hunting): authenticate will time out at 1414.589166s 2024-03-29T11:27:58.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.322+0000 7f9ded918640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9de80af900 0x7f9de8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:36357/0 (socket says 172.21.15.64:36357) 2024-03-29T11:27:58.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.322+0000 7f9ded918640 1 -- 172.21.15.64:0/2191893328 learned_addr learned my addr 172.21.15.64:0/2191893328 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:58.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.323+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660179593 0 0) 0x7f9de80af6f0 con 0x7f9de80af900 2024-03-29T11:27:58.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.323+0000 7f9ddffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.323+0000 7f9ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.323+0000 7f9ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.325+0000 7f9ddffff640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:27:58.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.325+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4003520 con 0x7f9de80af900 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.325+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083087242 0 0) 0x7f9de80afd10 con 0x7f9de80a9ac0 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.325+0000 7f9ddffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.325+0000 7f9ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de80af6f0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919590130 0 0) 0x7f9de80aac80 con 0x7f9de813ee90 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de80afd10 con 0x7f9de813ee90 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370094118 0 0) 0x7f9dd4003520 con 0x7f9de80af900 2024-03-29T11:27:58.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 >> v1:172.21.15.64:6790/0 conn(0x7f9de813ee90 legacy=0x7f9de813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 >> v1:172.21.15.64:6789/0 conn(0x7f9de80a9ac0 legacy=0x7f9de80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 monclient: found mon.noname-c 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80aaf60 con 0x7f9de80af900 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9dd8001540 con 0x7f9de80af900 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: dump: 2024-03-29T11:27:58.319 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dd8001760 con 0x7f9de80af900 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.326+0000 7f9ddffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9ddffff640 1 -- 172.21.15.64:0/2191893328 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9dd8001af0 con 0x7f9de80af900 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9ddffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9ddffff640 10 monclient: dump: 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9def37b640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9def37b640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9def37b640 4 set_mon_vals no callback set 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9def37b640 10 monclient: shutdown 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9def37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.327+0000 7f9def37b640 1 -- 172.21.15.64:0/2191893328 >> v1:172.21.15.136:6789/0 conn(0x7f9de80af900 legacy=0x7f9de8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.328+0000 7f9def37b640 1 -- 172.21.15.64:0/2191893328 shutdown_connections 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.328+0000 7f9def37b640 1 -- 172.21.15.64:0/2191893328 >> 172.21.15.64:0/2191893328 conn(0x7f9de80b35e0 msgr2=0x7f9de80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.333+0000 7f9def37b640 1 -- 172.21.15.64:0/2191893328 shutdown_connections 2024-03-29T11:27:58.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.333+0000 7f9def37b640 1 -- 172.21.15.64:0/2191893328 wait complete. 2024-03-29T11:27:58.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.334+0000 7f9def37b640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.334+0000 7f9def37b640 10 monclient: monmap: 2024-03-29T11:27:58.323 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:27:58.324 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 1 librados: starting msgr at 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 1 librados: starting objecter 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 1 Processor -- start 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 1 -- start start 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 1 librados: setting wanted keys 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 1 librados: calling monclient init 2024-03-29T11:27:58.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.335+0000 7f9def37b640 10 monclient: init 2024-03-29T11:27:58.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.336+0000 7f9def37b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.336+0000 7f9def37b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.339+0000 7f9def37b640 10 monclient(hunting): picked mon.noname-b con 0x7f9de80a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T11:27:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.340+0000 7f9ded117640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9de80a9ac0 0x7f9de806ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:36431/0 (socket says 172.21.15.64:36431) 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.340+0000 7f9ded117640 1 -- 172.21.15.64:0/3741150882 learned_addr learned my addr 172.21.15.64:0/3741150882 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 10 monclient(hunting): picked mon.noname-c con 0x7f9de80af900 addr v1:172.21.15.64:6790/0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 10 monclient(hunting): picked mon.noname-a con 0x7f9de813ee90 addr v1:172.21.15.64:6789/0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80707f0 con 0x7f9de813ee90 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8142270 con 0x7f9de80a9ac0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de806f540 con 0x7f9de80af900 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.341+0000 7f9def37b640 10 monclient(hunting): authenticate will time out at 1414.611128s 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dddffb640 10 client.?.objecter ms_handle_connect 0x7f9de80a9ac0 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dec916640 1 -- 172.21.15.64:0/3741150882 >> v1:172.21.15.64:6790/0 conn(0x7f9de80af900 legacy=0x7f9de81dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dec916640 1 -- 172.21.15.64:0/3741150882 >> v1:172.21.15.64:6790/0 conn(0x7f9de80af900 legacy=0x7f9de81dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dec916640 1 --1- 172.21.15.64:0/3741150882 >> v1:172.21.15.64:6790/0 conn(0x7f9de80af900 0x7f9de81dbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dec916640 1 --1- 172.21.15.64:0/3741150882 >> v1:172.21.15.64:6790/0 conn(0x7f9de80af900 0x7f9de81dbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.343+0000 7f9dddffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6790/0 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.344+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174385725 0 0) 0x7f9de8142270 con 0x7f9de80a9ac0 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.344+0000 7f9dddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.344+0000 7f9dddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.344+0000 7f9dddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.344+0000 7f9dddffb640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.344+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc004eb0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 10 client.?.objecter ms_handle_connect 0x7f9de813ee90 2024-03-29T11:27:58.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389884415 0 0) 0x7f9de80707f0 con 0x7f9de813ee90 2024-03-29T11:27:58.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.347+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8142270 con 0x7f9de813ee90 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1794381739 0 0) 0x7f9dcc004eb0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9de80707f0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9de0001990 con 0x7f9de80a9ac0 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 10 monclient(hunting): dump: 2024-03-29T11:27:58.338 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:58.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089897247 0 0) 0x7f9de80707f0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 >> v1:172.21.15.64:6790/0 conn(0x7f9de80af900 legacy=0x7f9de81dbd60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 >> v1:172.21.15.64:6789/0 conn(0x7f9de813ee90 legacy=0x7f9de8070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 monclient: found mon.b 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.348+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de806f120 con 0x7f9de80a9ac0 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de0001af0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f9de0003a60 con 0x7f9de80a9ac0 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9dddffb640 10 monclient: dump: 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:27:58.341 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.349+0000 7f9ddf7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.351+0000 7f9def37b640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.351+0000 7f9def37b640 10 monclient: _renew_subs 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.351+0000 7f9def37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.351+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de81dc470 con 0x7f9de80a9ac0 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.352+0000 7f9def37b640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.352+0000 7f9def37b640 10 monclient: _renew_subs 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.352+0000 7f9def37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.352+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de8071500 con 0x7f9de80a9ac0 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.352+0000 7f9def37b640 1 librados: init done 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.355+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f9de0003bc0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dec916640 10 monclient: get_auth_request con 0x7f9dcc06dd00 auth_method 0 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9def37b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9def37b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9def37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de806f330 con 0x7f9de80a9ac0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f9de0084af0 con 0x7f9de80a9ac0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dddffb640 10 client.4390.objecter ms_dispatch 0x7f9de8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dddffb640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dddffb640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dddffb640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.356+0000 7f9dddffb640 10 client.4390.objecter ms_handle_connect 0x7f9dcc06dd00 2024-03-29T11:27:58.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.366+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9de806f330 con 0x7f9de80a9ac0 2024-03-29T11:27:58.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.366+0000 7f9dddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.366+0000 7f9dddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:58.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.699+0000 7f9def37b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.699+0000 7f9def37b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.699+0000 7f9def37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:27:58.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.699+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9de806f330 con 0x7f9de80a9ac0 2024-03-29T11:27:58.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.700+0000 7f9dddffb640 1 -- 172.21.15.64:0/3741150882 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f9de806f330 con 0x7f9de80a9ac0 2024-03-29T11:27:58.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.700+0000 7f9dddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.700+0000 7f9dddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:58.690 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:27:58.690 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:58.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.704+0000 7f9def37b640 10 librados: watch_flush enter 2024-03-29T11:27:58.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 10 librados: watch_flush exit 2024-03-29T11:27:58.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:27:58.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:27:58.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 >> v1:172.21.15.136:6816/2312274743 conn(0x7f9dcc06dd00 legacy=0x7f9dcc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 10 monclient: shutdown 2024-03-29T11:27:58.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.706+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 >> v1:172.21.15.136:6789/0 conn(0x7f9de80a9ac0 legacy=0x7f9de806ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.707+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 shutdown_connections 2024-03-29T11:27:58.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.707+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 >> 172.21.15.64:0/3741150882 conn(0x7f9de80b35e0 msgr2=0x7f9de80ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.708+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 shutdown_connections 2024-03-29T11:27:58.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.708+0000 7f9def37b640 1 -- 172.21.15.64:0/3741150882 wait complete. 2024-03-29T11:27:58.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:27:58.709+0000 7f9def37b640 1 librados: shutdown 2024-03-29T11:27:58.715 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:05.523 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:05.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.688+0000 7f757b756640 10 monclient: get_monmap_and_config 2024-03-29T11:28:05.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.688+0000 7f757b756640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.688+0000 7f757b756640 10 monclient: monmap: 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:05.689803+0000 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:05.689803+0000 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:05.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.689+0000 7f757b756640 1 Processor -- start 2024-03-29T11:28:05.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.689+0000 7f757b756640 1 -- start start 2024-03-29T11:28:05.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.689+0000 7f757b756640 10 monclient: init 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient(hunting): picked mon.noname-c con 0x7f75740b0770 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient(hunting): picked mon.noname-a con 0x7f757414a700 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient(hunting): picked mon.noname-b con 0x7f75740ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740aba70 con 0x7f757414a700 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740aecb0 con 0x7f75740b0770 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740af150 con 0x7f75740ac5a0 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.690+0000 7f757b756640 10 monclient(hunting): authenticate will time out at 1421.960788s 2024-03-29T11:28:05.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.693+0000 7f7578cf1640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f757414a700 0x7f75740abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40615/0 (socket says 172.21.15.64:40615) 2024-03-29T11:28:05.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.693+0000 7f7578cf1640 1 -- 172.21.15.64:0/1180155042 learned_addr learned my addr 172.21.15.64:0/1180155042 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.696+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555613247 0 0) 0x7f75740aba70 con 0x7f757414a700 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.696+0000 7f7567fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.696+0000 7f7567fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.696+0000 7f7567fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.696+0000 7f7567fff640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.696+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7560003520 con 0x7f757414a700 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810201954 0 0) 0x7f7560003520 con 0x7f757414a700 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 >> v1:172.21.15.64:6790/0 conn(0x7f75740ac5a0 legacy=0x7f7574151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 >> v1:172.21.15.136:6789/0 conn(0x7f75740b0770 legacy=0x7f7574149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 1 monclient: found mon.noname-a 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75740ac9d0 con 0x7f757414a700 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f756c001a10 con 0x7f757414a700 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f7567fff640 10 monclient: dump: 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.690 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f757b756640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.697+0000 7f757b756640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f756c001c30 con 0x7f757414a700 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f7567fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f7567fff640 1 -- 172.21.15.64:0/1180155042 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f756c001fc0 con 0x7f757414a700 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f7567fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f7567fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f7567fff640 10 monclient: dump: 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:05.691 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:05.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f757b756640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.698+0000 7f757b756640 4 set_mon_vals no callback set 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.699+0000 7f757b756640 10 monclient: shutdown 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.699+0000 7f757b756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.699+0000 7f757b756640 1 -- 172.21.15.64:0/1180155042 >> v1:172.21.15.64:6789/0 conn(0x7f757414a700 legacy=0x7f75740abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.699+0000 7f757b756640 1 -- 172.21.15.64:0/1180155042 shutdown_connections 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.699+0000 7f757b756640 1 -- 172.21.15.64:0/1180155042 >> 172.21.15.64:0/1180155042 conn(0x7f757406b660 msgr2=0x7f757406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.699+0000 7f757b756640 1 -- 172.21.15.64:0/1180155042 shutdown_connections 2024-03-29T11:28:05.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.701+0000 7f757b756640 1 -- 172.21.15.64:0/1180155042 wait complete. 2024-03-29T11:28:05.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.702+0000 7f757b756640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.702+0000 7f757b756640 10 monclient: monmap: 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.702+0000 7f757b756640 1 librados: starting msgr at 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.702+0000 7f757b756640 1 librados: starting objecter 2024-03-29T11:28:05.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.702+0000 7f757b756640 1 Processor -- start 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 1 -- start start 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 1 librados: setting wanted keys 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 1 librados: calling monclient init 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient: init 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient(hunting): picked mon.noname-a con 0x7f75740ac5a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient(hunting): picked mon.noname-c con 0x7f75740b0770 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient(hunting): picked mon.noname-b con 0x7f757414a700 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75741db5a0 con 0x7f75740ac5a0 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757412a400 con 0x7f757414a700 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75741dc420 con 0x7f75740b0770 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.703+0000 7f757b756640 10 monclient(hunting): authenticate will time out at 1421.973683s 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.708+0000 7f7579cf3640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f757414a700 0x7f75741dbd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:55859/0 (socket says 172.21.15.64:55859) 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.708+0000 7f7579cf3640 1 -- 172.21.15.64:0/2661360010 learned_addr learned my addr 172.21.15.64:0/2661360010 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 client.?.objecter ms_handle_connect 0x7f757414a700 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 client.?.objecter ms_handle_connect 0x7f75740ac5a0 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432283416 0 0) 0x7f757412a400 con 0x7f757414a700 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7558004eb0 con 0x7f757414a700 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281424495 0 0) 0x7f75741db5a0 con 0x7f75740ac5a0 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.709+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75580051d0 con 0x7f75740ac5a0 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378051889 0 0) 0x7f75580051d0 con 0x7f75740ac5a0 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7558005640 con 0x7f75740ac5a0 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7570001860 con 0x7f75740ac5a0 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.704 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:05.705 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:05.705 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:05.705 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:05.705 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:05.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.710+0000 7f7565ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:05.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 client.?.objecter ms_handle_connect 0x7f75740b0770 2024-03-29T11:28:05.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2799192006 0 0) 0x7f7558005640 con 0x7f75740ac5a0 2024-03-29T11:28:05.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 >> v1:172.21.15.64:6790/0 conn(0x7f75740b0770 legacy=0x7f7574127f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 >> v1:172.21.15.136:6789/0 conn(0x7f757414a700 legacy=0x7f75741dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 monclient: found mon.a 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574125440 con 0x7f75740ac5a0 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75700035f0 con 0x7f75740ac5a0 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7570001860 con 0x7f75740ac5a0 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f7565ffb640 10 monclient: dump: 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:05.708 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.712+0000 7f75677fe640 4 set_mon_vals no callback set 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.713+0000 7f757b756640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:28:05.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.713+0000 7f757b756640 10 monclient: _renew_subs 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.713+0000 7f757b756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.713+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7574125690 con 0x7f75740ac5a0 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.714+0000 7f757b756640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.714+0000 7f757b756640 10 monclient: _renew_subs 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.714+0000 7f757b756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.714+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75741e72b0 con 0x7f75740ac5a0 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.714+0000 7f757b756640 1 librados: init done 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.716+0000 7f757b756640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.716+0000 7f757b756640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.716+0000 7f757b756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:05.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.716+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75740ab560 con 0x7f75740ac5a0 2024-03-29T11:28:05.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.718+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f75700195d0 con 0x7f75740ac5a0 2024-03-29T11:28:05.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.719+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f7570083210 con 0x7f75740ac5a0 2024-03-29T11:28:05.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.719+0000 7f7565ffb640 10 client.4374.objecter ms_dispatch 0x7f7574000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:05.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.719+0000 7f7565ffb640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:05.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.719+0000 7f7565ffb640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:05.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.720+0000 7f7565ffb640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:28:05.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.720+0000 7f7578cf1640 10 monclient: get_auth_request con 0x7f755806e210 auth_method 0 2024-03-29T11:28:05.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.723+0000 7f7565ffb640 10 client.4374.objecter ms_handle_connect 0x7f755806e210 2024-03-29T11:28:05.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.727+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f757004ea60 con 0x7f75740ac5a0 2024-03-29T11:28:05.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.727+0000 7f7565ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:05.727+0000 7f7565ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.053+0000 7f757b756640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.053+0000 7f757b756640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.053+0000 7f757b756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.053+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7574075f10 con 0x7f75740ac5a0 2024-03-29T11:28:06.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.055+0000 7f7565ffb640 1 -- 172.21.15.64:0/2661360010 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f757004ea60 con 0x7f75740ac5a0 2024-03-29T11:28:06.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.055+0000 7f7565ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:06.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.055+0000 7f7565ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:06.045 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:06.045 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:06.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.059+0000 7f757b756640 10 librados: watch_flush enter 2024-03-29T11:28:06.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.059+0000 7f757b756640 10 librados: watch_flush exit 2024-03-29T11:28:06.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.059+0000 7f757b756640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:28:06.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.059+0000 7f757b756640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:28:06.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 >> v1:172.21.15.136:6816/2312274743 conn(0x7f755806e210 legacy=0x7f75580706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 10 monclient: shutdown 2024-03-29T11:28:06.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:06.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 >> v1:172.21.15.64:6789/0 conn(0x7f75740ac5a0 legacy=0x7f7574128e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 shutdown_connections 2024-03-29T11:28:06.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 >> 172.21.15.64:0/2661360010 conn(0x7f757406b660 msgr2=0x7f75740abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:06.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 shutdown_connections 2024-03-29T11:28:06.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 -- 172.21.15.64:0/2661360010 wait complete. 2024-03-29T11:28:06.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:06.062+0000 7f757b756640 1 librados: shutdown 2024-03-29T11:28:06.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:12.877 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.042+0000 7f1b665ad640 10 monclient: get_monmap_and_config 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.042+0000 7f1b665ad640 10 monclient: build_initial_monmap 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.042+0000 7f1b665ad640 10 monclient: monmap: 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:13.043300+0000 2024-03-29T11:28:13.034 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:13.043300+0000 2024-03-29T11:28:13.035 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:13.035 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:13.035 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:13.035 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:13.035 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:13.035 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:13.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.046+0000 7f1b665ad640 1 Processor -- start 2024-03-29T11:28:13.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.047+0000 7f1b665ad640 1 -- start start 2024-03-29T11:28:13.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.047+0000 7f1b665ad640 10 monclient: init 2024-03-29T11:28:13.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.048+0000 7f1b665ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:13.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.048+0000 7f1b665ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:13.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.048+0000 7f1b665ad640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6014a400 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:13.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.048+0000 7f1b665ad640 10 monclient(hunting): picked mon.noname-a con 0x7f1b600aa500 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:13.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.051+0000 7f1b64daa640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1b600aa500 0x7f1b600ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59133/0 (socket says 172.21.15.64:59133) 2024-03-29T11:28:13.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b665ad640 10 monclient(hunting): picked mon.noname-c con 0x7f1b60149800 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:13.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b665ad640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b600aca50 con 0x7f1b600aa500 2024-03-29T11:28:13.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b665ad640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b60149cb0 con 0x7f1b60149800 2024-03-29T11:28:13.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b665ad640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b600acc80 con 0x7f1b6014a400 2024-03-29T11:28:13.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b665ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:13.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b665ad640 10 monclient(hunting): authenticate will time out at 1429.321957s 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.052+0000 7f1b64daa640 1 -- 172.21.15.64:0/4044817547 learned_addr learned my addr 172.21.15.64:0/4044817547 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.054+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740261853 0 0) 0x7f1b600aca50 con 0x7f1b600aa500 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.054+0000 7f1b57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.054+0000 7f1b57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.054+0000 7f1b57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.054+0000 7f1b57fff640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.054+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b4c003520 con 0x7f1b600aa500 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.057+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294776628 0 0) 0x7f1b60149cb0 con 0x7f1b60149800 2024-03-29T11:28:13.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.057+0000 7f1b57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.057+0000 7f1b57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.057+0000 7f1b57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.057+0000 7f1b57fff640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.057+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b600aca50 con 0x7f1b60149800 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620697877 0 0) 0x7f1b600aca50 con 0x7f1b60149800 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 >> v1:172.21.15.64:6790/0 conn(0x7f1b6014a400 legacy=0x7f1b600aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 >> v1:172.21.15.64:6789/0 conn(0x7f1b600aa500 legacy=0x7f1b600ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 monclient: found mon.noname-c 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b600ac350 con 0x7f1b60149800 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b665ad640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b665ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b50001670 con 0x7f1b60149800 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: dump: 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:13.052 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b665ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b50001890 con 0x7f1b60149800 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b665ad640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.058+0000 7f1b665ad640 4 set_mon_vals no callback set 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b57fff640 1 -- 172.21.15.64:0/4044817547 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b50001c20 con 0x7f1b60149800 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b57fff640 10 monclient: dump: 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b665ad640 10 monclient: shutdown 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b665ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:13.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b665ad640 1 -- 172.21.15.64:0/4044817547 >> v1:172.21.15.136:6789/0 conn(0x7f1b60149800 legacy=0x7f1b60151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b665ad640 1 -- 172.21.15.64:0/4044817547 shutdown_connections 2024-03-29T11:28:13.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.059+0000 7f1b665ad640 1 -- 172.21.15.64:0/4044817547 >> 172.21.15.64:0/4044817547 conn(0x7f1b600b3a30 msgr2=0x7f1b600b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:13.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.060+0000 7f1b665ad640 1 -- 172.21.15.64:0/4044817547 shutdown_connections 2024-03-29T11:28:13.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.060+0000 7f1b665ad640 1 -- 172.21.15.64:0/4044817547 wait complete. 2024-03-29T11:28:13.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 10 monclient: build_initial_monmap 2024-03-29T11:28:13.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 10 monclient: monmap: 2024-03-29T11:28:13.056 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 1 librados: starting msgr at 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 1 librados: starting objecter 2024-03-29T11:28:13.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 1 Processor -- start 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 1 -- start start 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 1 librados: setting wanted keys 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 1 librados: calling monclient init 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.061+0000 7f1b665ad640 10 monclient: init 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.062+0000 7f1b665ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.062+0000 7f1b665ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.062+0000 7f1b665ad640 10 monclient(hunting): picked mon.noname-c con 0x7f1b600aa500 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:13.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.065+0000 7f1b665ad640 10 monclient(hunting): picked mon.noname-b con 0x7f1b60149800 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b665ad640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6014a400 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b665ad640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6014aba0 con 0x7f1b6014a400 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b665ad640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6013bba0 con 0x7f1b60149800 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b665ad640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6014ba70 con 0x7f1b600aa500 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b665ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b665ad640 10 monclient(hunting): authenticate will time out at 1429.338398s 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b64daa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1b60149800 0x7f1b6013ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:41223/0 (socket says 172.21.15.64:41223) 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b64daa640 1 -- 172.21.15.64:0/569095476 learned_addr learned my addr 172.21.15.64:0/569095476 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:13.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b55ffb640 10 client.?.objecter ms_handle_connect 0x7f1b60149800 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b55ffb640 10 client.?.objecter ms_handle_connect 0x7f1b6014a400 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.068+0000 7f1b55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991499484 0 0) 0x7f1b6013bba0 con 0x7f1b60149800 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b44004eb0 con 0x7f1b60149800 2024-03-29T11:28:13.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2640498286 0 0) 0x7f1b44004eb0 con 0x7f1b60149800 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b6013bba0 con 0x7f1b60149800 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b58001cd0 con 0x7f1b60149800 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.065 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:13.066 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:13.066 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:13.068 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:13.069 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:13.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.069+0000 7f1b55ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:13.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.070+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46194539 0 0) 0x7f1b6013bba0 con 0x7f1b60149800 2024-03-29T11:28:13.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.070+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 >> v1:172.21.15.64:6790/0 conn(0x7f1b600aa500 legacy=0x7f1b6013ec40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:13.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 >> v1:172.21.15.64:6789/0 conn(0x7f1b6014a400 legacy=0x7f1b6013b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 1 monclient: found mon.b 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6014bd50 con 0x7f1b60149800 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 10 monclient: _renew_subs 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:13.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6014a7e0 con 0x7f1b60149800 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 10 monclient: _renew_subs 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6013e810 con 0x7f1b60149800 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b665ad640 1 librados: init done 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b58004b60 con 0x7f1b60149800 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1b58004ef0 con 0x7f1b60149800 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.071+0000 7f1b55ffb640 10 monclient: dump: 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:13.071 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:13.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.073+0000 7f1b577fe640 4 set_mon_vals no callback set 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.074+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f1b58005050 con 0x7f1b60149800 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.075+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f1b58084cf0 con 0x7f1b60149800 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.075+0000 7f1b55ffb640 10 client.4399.objecter ms_dispatch 0x7f1b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.075+0000 7f1b55ffb640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.075+0000 7f1b55ffb640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.075+0000 7f1b55ffb640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.076+0000 7f1b655ab640 10 monclient: get_auth_request con 0x7f1b4406dcd0 auth_method 0 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.076+0000 7f1b665ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.076+0000 7f1b665ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.076+0000 7f1b665ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.077+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b40005080 con 0x7f1b60149800 2024-03-29T11:28:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.077+0000 7f1b55ffb640 10 client.4399.objecter ms_handle_connect 0x7f1b4406dcd0 2024-03-29T11:28:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.083+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b580505d0 con 0x7f1b60149800 2024-03-29T11:28:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.083+0000 7f1b55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.083+0000 7f1b55ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:13.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.424+0000 7f1b665ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.424+0000 7f1b665ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.424+0000 7f1b665ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:13.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.425+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b40005330 con 0x7f1b60149800 2024-03-29T11:28:13.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.426+0000 7f1b55ffb640 1 -- 172.21.15.64:0/569095476 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f1b58052500 con 0x7f1b60149800 2024-03-29T11:28:13.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.426+0000 7f1b55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.426+0000 7f1b55ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:13.417 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:13.417 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:13.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 10 librados: watch_flush enter 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 10 librados: watch_flush exit 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 10 client.4399.objecter successfully canceled tick 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 >> v1:172.21.15.136:6816/2312274743 conn(0x7f1b4406dcd0 legacy=0x7f1b44070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 10 monclient: shutdown 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.431+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 >> v1:172.21.15.136:6789/0 conn(0x7f1b60149800 legacy=0x7f1b6013ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.432+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 shutdown_connections 2024-03-29T11:28:13.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.432+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 >> 172.21.15.64:0/569095476 conn(0x7f1b600b3a30 msgr2=0x7f1b60069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:13.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.432+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 shutdown_connections 2024-03-29T11:28:13.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.432+0000 7f1b665ad640 1 -- 172.21.15.64:0/569095476 wait complete. 2024-03-29T11:28:13.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:13.433+0000 7f1b665ad640 1 librados: shutdown 2024-03-29T11:28:13.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:20.247 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:20.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.419+0000 7fd26a41e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:20.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.420+0000 7fd26a41e640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.420+0000 7fd26a41e640 10 monclient: monmap: 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:20.420995+0000 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:20.420995+0000 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:20.410 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:20.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.421+0000 7fd26a41e640 1 Processor -- start 2024-03-29T11:28:20.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.421+0000 7fd26a41e640 1 -- start start 2024-03-29T11:28:20.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.421+0000 7fd26a41e640 10 monclient: init 2024-03-29T11:28:20.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.422+0000 7fd26a41e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.422+0000 7fd26a41e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.422+0000 7fd26a41e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2641457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 10 monclient(hunting): picked mon.noname-a con 0x7fd26413f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2640a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd2689bb640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd2640a9d70 0x7fd26414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49034/0 (socket says 172.21.15.64:49034) 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd2689bb640 1 -- 172.21.15.64:0/45762339 learned_addr learned my addr 172.21.15.64:0/45762339 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2640aaef0 con 0x7fd26413f650 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd264145bd0 con 0x7fd2641457f0 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd264145eb0 con 0x7fd2640a9d70 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.423+0000 7fd26a41e640 10 monclient(hunting): authenticate will time out at 1436.693443s 2024-03-29T11:28:20.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912967447 0 0) 0x7fd264145bd0 con 0x7fd2641457f0 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24c003520 con 0x7fd2641457f0 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636148889 0 0) 0x7fd2640aaef0 con 0x7fd26413f650 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd264145bd0 con 0x7fd26413f650 2024-03-29T11:28:20.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005977707 0 0) 0x7fd264145eb0 con 0x7fd2640a9d70 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2640aaef0 con 0x7fd2640a9d70 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 493368066 0 0) 0x7fd24c003520 con 0x7fd2641457f0 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 >> v1:172.21.15.64:6790/0 conn(0x7fd2640a9d70 legacy=0x7fd26414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 >> v1:172.21.15.64:6789/0 conn(0x7fd26413f650 legacy=0x7fd2640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 monclient: found mon.noname-c 2024-03-29T11:28:20.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264146340 con 0x7fd2641457f0 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd258001690 con 0x7fd2641457f0 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd262ffd640 10 monclient: dump: 2024-03-29T11:28:20.418 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd26a41e640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.424+0000 7fd26a41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2580017f0 con 0x7fd2641457f0 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd262ffd640 1 -- 172.21.15.64:0/45762339 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd258001c00 con 0x7fd2641457f0 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd262ffd640 10 monclient: dump: 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd26a41e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:20.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.425+0000 7fd26a41e640 4 set_mon_vals no callback set 2024-03-29T11:28:20.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.426+0000 7fd26a41e640 10 monclient: shutdown 2024-03-29T11:28:20.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.426+0000 7fd26a41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.426+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 >> v1:172.21.15.136:6789/0 conn(0x7fd2641457f0 legacy=0x7fd26413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.426+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 shutdown_connections 2024-03-29T11:28:20.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.426+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 >> 172.21.15.64:0/45762339 conn(0x7fd264066a00 msgr2=0x7fd264066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.427+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 shutdown_connections 2024-03-29T11:28:20.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.428+0000 7fd26a41e640 1 -- 172.21.15.64:0/45762339 wait complete. 2024-03-29T11:28:20.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.429+0000 7fd26a41e640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.429+0000 7fd26a41e640 10 monclient: monmap: 2024-03-29T11:28:20.425 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 1 librados: starting msgr at 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 1 librados: starting objecter 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 1 Processor -- start 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 1 -- start start 2024-03-29T11:28:20.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 1 librados: setting wanted keys 2024-03-29T11:28:20.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 1 librados: calling monclient init 2024-03-29T11:28:20.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.430+0000 7fd26a41e640 10 monclient: init 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.431+0000 7fd26a41e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.431+0000 7fd26a41e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.431+0000 7fd26a41e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2640a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.431+0000 7fd26a41e640 10 monclient(hunting): picked mon.noname-b con 0x7fd26413f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.431+0000 7fd26a41e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2641457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd26a41e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26413b1b0 con 0x7fd2640a9d70 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd2637fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd26413f650 0x7fd264134fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:55176/0 (socket says 172.21.15.64:55176) 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd2637fe640 1 -- 172.21.15.64:0/175286533 learned_addr learned my addr 172.21.15.64:0/175286533 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:20.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd26a41e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26413b9a0 con 0x7fd26413f650 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd264135bf0 con 0x7fd2641457f0 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd26a41e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd26a41e640 10 monclient(hunting): authenticate will time out at 1436.702170s 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 client.?.objecter ms_handle_connect 0x7fd26413f650 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209714783 0 0) 0x7fd26413b9a0 con 0x7fd26413f650 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd250004eb0 con 0x7fd26413f650 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 client.?.objecter ms_handle_connect 0x7fd2641457f0 2024-03-29T11:28:20.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.432+0000 7fd260ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194994373 0 0) 0x7fd250004eb0 con 0x7fd26413f650 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd250005300 con 0x7fd26413f650 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd2540019f0 con 0x7fd26413f650 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.432 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 client.?.objecter ms_handle_connect 0x7fd2640a9d70 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615720579 0 0) 0x7fd264135bf0 con 0x7fd2641457f0 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd250005850 con 0x7fd2641457f0 2024-03-29T11:28:20.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4100294561 0 0) 0x7fd250005300 con 0x7fd26413f650 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 >> v1:172.21.15.64:6790/0 conn(0x7fd2641457f0 legacy=0x7fd264140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 >> v1:172.21.15.64:6789/0 conn(0x7fd2640a9d70 legacy=0x7fd264139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 monclient: found mon.b 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264138cc0 con 0x7fd26413f650 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.433+0000 7fd260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:28:20.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 10 monclient: _renew_subs 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26413a170 con 0x7fd26413f650 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 10 monclient: _renew_subs 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2641358c0 con 0x7fd26413f650 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd26a41e640 1 librados: init done 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd254001cd0 con 0x7fd26413f650 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd260ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd2540019f0 con 0x7fd26413f650 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd260ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:20.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd260ff9640 10 monclient: dump: 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.434+0000 7fd2627fc640 4 set_mon_vals no callback set 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.435+0000 7fd26a41e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.435+0000 7fd26a41e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.435+0000 7fd26a41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:20.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.435+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd244005080 con 0x7fd26413f650 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.436+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fd2540038c0 con 0x7fd26413f650 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.437+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fd254084620 con 0x7fd26413f650 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.437+0000 7fd260ff9640 10 client.4405.objecter ms_dispatch 0x7fd264000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.437+0000 7fd260ff9640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.437+0000 7fd260ff9640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.437+0000 7fd260ff9640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.437+0000 7fd263fff640 10 monclient: get_auth_request con 0x7fd25006e580 auth_method 0 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.438+0000 7fd260ff9640 10 client.4405.objecter ms_handle_connect 0x7fd25006e580 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.443+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd25404ff00 con 0x7fd26413f650 2024-03-29T11:28:20.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.444+0000 7fd260ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.444+0000 7fd260ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:20.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.854+0000 7fd26a41e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.854+0000 7fd26a41e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.854+0000 7fd26a41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:20.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.854+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd244005330 con 0x7fd26413f650 2024-03-29T11:28:20.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.855+0000 7fd260ff9640 1 -- 172.21.15.64:0/175286533 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fd25404ff00 con 0x7fd26413f650 2024-03-29T11:28:20.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.855+0000 7fd260ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.855+0000 7fd260ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:20.845 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:20.845 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 10 librados: watch_flush enter 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 10 librados: watch_flush exit 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 >> v1:172.21.15.136:6816/2312274743 conn(0x7fd25006e580 legacy=0x7fd250070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 10 monclient: shutdown 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.859+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 >> v1:172.21.15.136:6789/0 conn(0x7fd26413f650 legacy=0x7fd264134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.860+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 shutdown_connections 2024-03-29T11:28:20.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.860+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 >> 172.21.15.64:0/175286533 conn(0x7fd264066a00 msgr2=0x7fd2640b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.860+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 shutdown_connections 2024-03-29T11:28:20.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.861+0000 7fd26a41e640 1 -- 172.21.15.64:0/175286533 wait complete. 2024-03-29T11:28:20.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:20.861+0000 7fd26a41e640 1 librados: shutdown 2024-03-29T11:28:20.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:27.677 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:27.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.839+0000 7f1d629e8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:27.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.839+0000 7f1d629e8640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.839+0000 7f1d629e8640 10 monclient: monmap: 2024-03-29T11:28:27.829 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:27.829 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:27.829 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:27.840529+0000 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:27.840529+0000 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:27.830 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.840+0000 7f1d629e8640 1 Processor -- start 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.840+0000 7f1d629e8640 1 -- start start 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.840+0000 7f1d629e8640 10 monclient: init 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c0b2800 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c142b20 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c13c040 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0aad10 con 0x7f1d5c142b20 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c141f30 con 0x7f1d5c13c040 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d619e6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1d5c0b2800 0x7f1d5c142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:44818/0 (socket says 172.21.15.64:44818) 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d619e6640 1 -- 172.21.15.64:0/2282629510 learned_addr learned my addr 172.21.15.64:0/2282629510 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:27.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 1 -- 172.21.15.64:0/2282629510 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0aaf60 con 0x7f1d5c0b2800 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.841+0000 7f1d629e8640 10 monclient(hunting): authenticate will time out at 1444.111750s 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398654111 0 0) 0x7f1d5c141f30 con 0x7f1d5c13c040 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d48003520 con 0x7f1d5c13c040 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137809835 0 0) 0x7f1d5c0aaf60 con 0x7f1d5c0b2800 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:28:27.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c141f30 con 0x7f1d5c0b2800 2024-03-29T11:28:27.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232999706 0 0) 0x7f1d48003520 con 0x7f1d5c13c040 2024-03-29T11:28:27.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 >> v1:172.21.15.64:6790/0 conn(0x7f1d5c0b2800 legacy=0x7f1d5c142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 >> v1:172.21.15.64:6789/0 conn(0x7f1d5c142b20 legacy=0x7f1d5c13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 monclient: found mon.noname-c 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0b3900 con 0x7f1d5c13c040 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1d4c001690 con 0x7f1d5c13c040 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: dump: 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:27.837 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d629e8640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d629e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:27.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d4c0017f0 con 0x7f1d5c13c040 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.842+0000 7f1d609e4640 1 -- 172.21.15.64:0/2282629510 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1d4c001c00 con 0x7f1d5c13c040 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d609e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d609e4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d609e4640 10 monclient: dump: 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:27.839 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d629e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d629e8640 4 set_mon_vals no callback set 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d629e8640 10 monclient: shutdown 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d629e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.843+0000 7f1d629e8640 1 -- 172.21.15.64:0/2282629510 >> v1:172.21.15.136:6789/0 conn(0x7f1d5c13c040 legacy=0x7f1d5c14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.844+0000 7f1d629e8640 1 -- 172.21.15.64:0/2282629510 shutdown_connections 2024-03-29T11:28:27.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.844+0000 7f1d629e8640 1 -- 172.21.15.64:0/2282629510 >> 172.21.15.64:0/2282629510 conn(0x7f1d5c06d0e0 msgr2=0x7f1d5c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.844+0000 7f1d629e8640 1 -- 172.21.15.64:0/2282629510 shutdown_connections 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.844+0000 7f1d629e8640 1 -- 172.21.15.64:0/2282629510 wait complete. 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 10 monclient: monmap: 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 1 librados: starting msgr at 2024-03-29T11:28:27.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 1 librados: starting objecter 2024-03-29T11:28:27.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 1 Processor -- start 2024-03-29T11:28:27.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 1 -- start start 2024-03-29T11:28:27.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 1 librados: setting wanted keys 2024-03-29T11:28:27.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 1 librados: calling monclient init 2024-03-29T11:28:27.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.845+0000 7f1d629e8640 10 monclient: init 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c0b2800 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c13c040 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c142b20 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c06e080 con 0x7f1d5c142b20 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c1dc170 con 0x7f1d5c13c040 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d619e6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1d5c0b2800 0x7f1d5c144e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:44956/0 (socket says 172.21.15.64:44956) 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d619e6640 1 -- 172.21.15.64:0/3618481278 learned_addr learned my addr 172.21.15.64:0/3618481278 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c06e2f0 con 0x7f1d5c0b2800 2024-03-29T11:28:27.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.846+0000 7f1d629e8640 10 monclient(hunting): authenticate will time out at 1444.116760s 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 client.?.objecter ms_handle_connect 0x7f1d5c142b20 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 client.?.objecter ms_handle_connect 0x7f1d5c0b2800 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 client.?.objecter ms_handle_connect 0x7f1d5c13c040 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364184244 0 0) 0x7f1d5c06e080 con 0x7f1d5c142b20 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40004eb0 con 0x7f1d5c142b20 2024-03-29T11:28:27.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564452377 0 0) 0x7f1d5c1dc170 con 0x7f1d5c13c040 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c06e080 con 0x7f1d5c13c040 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7607679 0 0) 0x7f1d5c06e2f0 con 0x7f1d5c0b2800 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.847+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c1dc170 con 0x7f1d5c0b2800 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703830463 0 0) 0x7f1d40004eb0 con 0x7f1d5c142b20 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d5c06e2f0 con 0x7f1d5c142b20 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1d4c001b40 con 0x7f1d5c142b20 2024-03-29T11:28:27.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): dump: 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:27.850 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224322605 0 0) 0x7f1d5c1dc170 con 0x7f1d5c0b2800 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d40004eb0 con 0x7f1d5c0b2800 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1d54001c30 con 0x7f1d5c0b2800 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient(hunting): dump: 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:27.851 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 136355482 0 0) 0x7f1d5c06e2f0 con 0x7f1d5c142b20 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 >> v1:172.21.15.64:6790/0 conn(0x7f1d5c0b2800 legacy=0x7f1d5c144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 >> v1:172.21.15.136:6789/0 conn(0x7f1d5c13c040 legacy=0x7f1d5c1db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 monclient: found mon.a 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:27.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c1de7e0 con 0x7f1d5c142b20 2024-03-29T11:28:27.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d629e8640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d629e8640 10 monclient: _renew_subs 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d629e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.848+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c0b13e0 con 0x7f1d5c142b20 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d629e8640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d629e8640 10 monclient: _renew_subs 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d629e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c13e3b0 con 0x7f1d5c142b20 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d629e8640 1 librados: init done 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d4c0038a0 con 0x7f1d5c142b20 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1d4c001250 con 0x7f1d5c142b20 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d527fc640 10 monclient: dump: 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.849+0000 7f1d537fe640 4 set_mon_vals no callback set 2024-03-29T11:28:27.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.850+0000 7f1d629e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.850+0000 7f1d629e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.850+0000 7f1d629e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.850+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d44005080 con 0x7f1d5c142b20 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.852+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f1d4c019a20 con 0x7f1d5c142b20 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.852+0000 7f1d619e6640 10 monclient: get_auth_request con 0x7f1d4006df40 auth_method 0 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.852+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f1d4c083900 con 0x7f1d5c142b20 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.852+0000 7f1d527fc640 10 client.4392.objecter ms_dispatch 0x7f1d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.852+0000 7f1d527fc640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.852+0000 7f1d527fc640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.853+0000 7f1d527fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:28:27.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.853+0000 7f1d527fc640 10 client.4392.objecter ms_handle_connect 0x7f1d4006df40 2024-03-29T11:28:27.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.860+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d4c04f010 con 0x7f1d5c142b20 2024-03-29T11:28:27.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.860+0000 7f1d527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:27.860+0000 7f1d527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:28.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.242+0000 7f1d629e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.242+0000 7f1d629e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.242+0000 7f1d629e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:28.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.242+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d44005330 con 0x7f1d5c142b20 2024-03-29T11:28:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.243+0000 7f1d527fc640 1 -- 172.21.15.64:0/3618481278 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f1d4c051110 con 0x7f1d5c142b20 2024-03-29T11:28:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.243+0000 7f1d527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.243+0000 7f1d527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:28.233 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:28.233 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 10 librados: watch_flush enter 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 10 librados: watch_flush exit 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 >> v1:172.21.15.136:6816/2312274743 conn(0x7f1d4006df40 legacy=0x7f1d40070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 10 monclient: shutdown 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:28.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.246+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 >> v1:172.21.15.64:6789/0 conn(0x7f1d5c142b20 legacy=0x7f1d5c1dba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.247+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 shutdown_connections 2024-03-29T11:28:28.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.247+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 >> 172.21.15.64:0/3618481278 conn(0x7f1d5c06d0e0 msgr2=0x7f1d5c0b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.247+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 shutdown_connections 2024-03-29T11:28:28.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.247+0000 7f1d629e8640 1 -- 172.21.15.64:0/3618481278 wait complete. 2024-03-29T11:28:28.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:28.247+0000 7f1d629e8640 1 librados: shutdown 2024-03-29T11:28:28.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:35.058 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:35.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.220+0000 7fc1e8b8d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.220+0000 7fc1e8b8d640 10 monclient: build_initial_monmap 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.220+0000 7fc1e8b8d640 10 monclient: monmap: 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:35.222534+0000 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:35.222534+0000 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:35.212 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:35.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.221+0000 7fc1e8b8d640 1 Processor -- start 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.221+0000 7fc1e8b8d640 1 -- start start 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.221+0000 7fc1e8b8d640 10 monclient: init 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.222+0000 7fc1e8b8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.222+0000 7fc1e8b8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.222+0000 7fc1e8b8d640 10 monclient(hunting): picked mon.noname-b con 0x7fc1e41422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.222+0000 7fc1e8b8d640 10 monclient(hunting): picked mon.noname-a con 0x7fc1e4142690 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e8b8d640 10 monclient(hunting): picked mon.noname-c con 0x7fc1e40b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc1e41422b0 0x7fc1e413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50740/0 (socket says 172.21.15.64:50740) 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e259c640 1 -- 172.21.15.64:0/1403149150 learned_addr learned my addr 172.21.15.64:0/1403149150 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e4142c80 con 0x7fc1e4142690 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e40aae40 con 0x7fc1e40b17d0 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e413fbd0 con 0x7fc1e41422b0 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e8b8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:35.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e8b8d640 10 monclient(hunting): authenticate will time out at 1451.494513s 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624097379 0 0) 0x7fc1e4142c80 con 0x7fc1e4142690 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e1d9b640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.223+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d0003520 con 0x7fc1e4142690 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839402008 0 0) 0x7fc1e413fbd0 con 0x7fc1e41422b0 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e4142c80 con 0x7fc1e41422b0 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147640722 0 0) 0x7fc1d0003520 con 0x7fc1e4142690 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 >> v1:172.21.15.64:6790/0 conn(0x7fc1e41422b0 legacy=0x7fc1e413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 >> v1:172.21.15.136:6789/0 conn(0x7fc1e40b17d0 legacy=0x7fc1e40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 monclient: found mon.noname-a 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e414e910 con 0x7fc1e4142690 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc1d4001690 con 0x7fc1e4142690 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.224+0000 7fc1e1d9b640 10 monclient: dump: 2024-03-29T11:28:35.218 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e8b8d640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:28:35.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e8b8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1d40018b0 con 0x7fc1e4142690 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e1d9b640 1 -- 172.21.15.64:0/1403149150 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc1d4001c40 con 0x7fc1e4142690 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e1d9b640 10 monclient: dump: 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:35.222 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e8b8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.225+0000 7fc1e8b8d640 4 set_mon_vals no callback set 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 10 monclient: shutdown 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 >> v1:172.21.15.64:6789/0 conn(0x7fc1e4142690 legacy=0x7fc1e40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 shutdown_connections 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 >> 172.21.15.64:0/1403149150 conn(0x7fc1e4066a00 msgr2=0x7fc1e4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 shutdown_connections 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.226+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/1403149150 wait complete. 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.228+0000 7fc1e8b8d640 10 monclient: build_initial_monmap 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.228+0000 7fc1e8b8d640 10 monclient: monmap: 2024-03-29T11:28:35.223 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.228+0000 7fc1e8b8d640 1 librados: starting msgr at 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.228+0000 7fc1e8b8d640 1 librados: starting objecter 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.229+0000 7fc1e8b8d640 1 Processor -- start 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.229+0000 7fc1e8b8d640 1 -- start start 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.229+0000 7fc1e8b8d640 1 librados: setting wanted keys 2024-03-29T11:28:35.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.229+0000 7fc1e8b8d640 1 librados: calling monclient init 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.229+0000 7fc1e8b8d640 10 monclient: init 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient(hunting): picked mon.noname-c con 0x7fc1e40b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient(hunting): picked mon.noname-a con 0x7fc1e41422b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient(hunting): picked mon.noname-b con 0x7fc1e4142690 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e40acb70 con 0x7fc1e41422b0 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e40ada10 con 0x7fc1e4142690 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e40adc80 con 0x7fc1e40b17d0 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e8b8d640 10 monclient(hunting): authenticate will time out at 1451.501566s 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc1e40b17d0 0x7fc1e40ac270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52438/0 (socket says 172.21.15.64:52438) 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.230+0000 7fc1e259c640 1 -- 172.21.15.64:0/4002892595 learned_addr learned my addr 172.21.15.64:0/4002892595 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 client.?.objecter ms_handle_connect 0x7fc1e41422b0 2024-03-29T11:28:35.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271933801 0 0) 0x7fc1e40acb70 con 0x7fc1e41422b0 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c4004eb0 con 0x7fc1e41422b0 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 client.?.objecter ms_handle_connect 0x7fc1e4142690 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 client.?.objecter ms_handle_connect 0x7fc1e40b17d0 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.231+0000 7fc1db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478094924 0 0) 0x7fc1c4004eb0 con 0x7fc1e41422b0 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1e40acb70 con 0x7fc1e41422b0 2024-03-29T11:28:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc1d4001bc0 con 0x7fc1e41422b0 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204346466 0 0) 0x7fc1e40adc80 con 0x7fc1e40b17d0 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c4004eb0 con 0x7fc1e40b17d0 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922978154 0 0) 0x7fc1e40ada10 con 0x7fc1e4142690 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e40adc80 con 0x7fc1e4142690 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.232+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3365137259 0 0) 0x7fc1e40acb70 con 0x7fc1e41422b0 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 >> v1:172.21.15.64:6790/0 conn(0x7fc1e40b17d0 legacy=0x7fc1e40ac270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 >> v1:172.21.15.136:6789/0 conn(0x7fc1e4142690 legacy=0x7fc1e41e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 1 monclient: found mon.a 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e40ae7b0 con 0x7fc1e41422b0 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 10 monclient: _renew_subs 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e40ab5d0 con 0x7fc1e41422b0 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 10 monclient: _renew_subs 2024-03-29T11:28:35.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e4134620 con 0x7fc1e41422b0 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e8b8d640 1 librados: init done 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1d40038a0 con 0x7fc1e41422b0 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1e159a640 4 set_mon_vals no callback set 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc1d4003bb0 con 0x7fc1e41422b0 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.233+0000 7fc1db7fe640 10 monclient: dump: 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.233 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.236+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fc1d4019b80 con 0x7fc1e41422b0 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.236+0000 7fc1e8b8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.236+0000 7fc1e8b8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.236+0000 7fc1e8b8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:35.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.236+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c8005080 con 0x7fc1e41422b0 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1e259c640 10 monclient: get_auth_request con 0x7fc1c406def0 auth_method 0 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fc1d4083820 con 0x7fc1e41422b0 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1db7fe640 10 client.4398.objecter ms_dispatch 0x7fc1e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1db7fe640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1db7fe640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1db7fe640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:28:35.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.237+0000 7fc1db7fe640 10 client.4398.objecter ms_handle_connect 0x7fc1c406def0 2024-03-29T11:28:35.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.246+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1d404f100 con 0x7fc1e41422b0 2024-03-29T11:28:35.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.246+0000 7fc1db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.246+0000 7fc1db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.597+0000 7fc1e8b8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.598+0000 7fc1e8b8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.598+0000 7fc1e8b8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:28:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.598+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1c8005330 con 0x7fc1e41422b0 2024-03-29T11:28:35.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.600+0000 7fc1db7fe640 1 -- 172.21.15.64:0/4002892595 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fc1d4051030 con 0x7fc1e41422b0 2024-03-29T11:28:35.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.600+0000 7fc1db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.600+0000 7fc1db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:35.591 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:35.591 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 10 librados: watch_flush enter 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 10 librados: watch_flush exit 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 >> v1:172.21.15.136:6816/2312274743 conn(0x7fc1c406def0 legacy=0x7fc1c40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 10 monclient: shutdown 2024-03-29T11:28:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 >> v1:172.21.15.64:6789/0 conn(0x7fc1e41422b0 legacy=0x7fc1e40ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 shutdown_connections 2024-03-29T11:28:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 >> 172.21.15.64:0/4002892595 conn(0x7fc1e4066a00 msgr2=0x7fc1e40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 shutdown_connections 2024-03-29T11:28:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 -- 172.21.15.64:0/4002892595 wait complete. 2024-03-29T11:28:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:35.603+0000 7fc1e8b8d640 1 librados: shutdown 2024-03-29T11:28:35.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:42.417 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:42.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.577+0000 7f70c4510640 10 monclient: get_monmap_and_config 2024-03-29T11:28:42.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.577+0000 7f70c4510640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.577+0000 7f70c4510640 10 monclient: monmap: 2024-03-29T11:28:42.568 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:42.568 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:42.578096+0000 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:42.578096+0000 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:42.569 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.579+0000 7f70c4510640 1 Processor -- start 2024-03-29T11:28:42.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.579+0000 7f70c4510640 1 -- start start 2024-03-29T11:28:42.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.579+0000 7f70c4510640 10 monclient: init 2024-03-29T11:28:42.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:42.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient(hunting): picked mon.noname-a con 0x7f70bc0a9ac0 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:42.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient(hunting): picked mon.noname-c con 0x7f70bc13ee90 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:42.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient(hunting): picked mon.noname-b con 0x7f70bc0af900 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:42.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0afd10 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0af6f0 con 0x7f70bc13ee90 2024-03-29T11:28:42.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0aac80 con 0x7f70bc0af900 2024-03-29T11:28:42.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.580+0000 7f70c4510640 10 monclient(hunting): authenticate will time out at 1458.850865s 2024-03-29T11:28:42.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.583+0000 7f70c2aad640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f70bc0af900 0x7f70bc149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59870/0 (socket says 172.21.15.64:59870) 2024-03-29T11:28:42.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.583+0000 7f70c2aad640 1 -- 172.21.15.64:0/2562708931 learned_addr learned my addr 172.21.15.64:0/2562708931 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:42.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657646261 0 0) 0x7f70bc0afd10 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a8003520 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602178068 0 0) 0x7f70bc0aac80 con 0x7f70bc0af900 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:28:42.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.584+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70bc0afd10 con 0x7f70bc0af900 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.585+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075202089 0 0) 0x7f70a8003520 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.585+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 >> v1:172.21.15.64:6790/0 conn(0x7f70bc0af900 legacy=0x7f70bc149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 >> v1:172.21.15.136:6789/0 conn(0x7f70bc13ee90 legacy=0x7f70bc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 1 monclient: found mon.noname-a 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc0aaf60 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: _finish_auth 0 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f70b8001690 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c12aa640 10 monclient: dump: 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.578 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c4510640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.586+0000 7f70c4510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:42.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b80017f0 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c12aa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c12aa640 1 -- 172.21.15.64:0/2562708931 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f70b8001c00 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c12aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c12aa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c12aa640 10 monclient: dump: 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:42.580 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c4510640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.587+0000 7f70c4510640 4 set_mon_vals no callback set 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.588+0000 7f70c4510640 10 monclient: shutdown 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.588+0000 7f70c4510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.588+0000 7f70c4510640 1 -- 172.21.15.64:0/2562708931 >> v1:172.21.15.64:6789/0 conn(0x7f70bc0a9ac0 legacy=0x7f70bc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.589+0000 7f70c4510640 1 -- 172.21.15.64:0/2562708931 shutdown_connections 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.589+0000 7f70c4510640 1 -- 172.21.15.64:0/2562708931 >> 172.21.15.64:0/2562708931 conn(0x7f70bc0b35e0 msgr2=0x7f70bc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.591+0000 7f70c4510640 1 -- 172.21.15.64:0/2562708931 shutdown_connections 2024-03-29T11:28:42.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.591+0000 7f70c4510640 1 -- 172.21.15.64:0/2562708931 wait complete. 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.593+0000 7f70c4510640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.593+0000 7f70c4510640 10 monclient: monmap: 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.583 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:42.584 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:42.584 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:42.584 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.593+0000 7f70c4510640 1 librados: starting msgr at 2024-03-29T11:28:42.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.593+0000 7f70c4510640 1 librados: starting objecter 2024-03-29T11:28:42.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.593+0000 7f70c4510640 1 Processor -- start 2024-03-29T11:28:42.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.595+0000 7f70c4510640 1 -- start start 2024-03-29T11:28:42.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.595+0000 7f70c4510640 1 librados: setting wanted keys 2024-03-29T11:28:42.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.595+0000 7f70c4510640 1 librados: calling monclient init 2024-03-29T11:28:42.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.595+0000 7f70c4510640 10 monclient: init 2024-03-29T11:28:42.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:42.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient(hunting): picked mon.noname-b con 0x7f70bc0a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:42.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient(hunting): picked mon.noname-a con 0x7f70bc0af900 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:42.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient(hunting): picked mon.noname-c con 0x7f70bc13ee90 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:42.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc071c40 con 0x7f70bc0af900 2024-03-29T11:28:42.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc071480 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0718a0 con 0x7f70bc13ee90 2024-03-29T11:28:42.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.596+0000 7f70c4510640 10 monclient(hunting): authenticate will time out at 1458.866682s 2024-03-29T11:28:42.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.599+0000 7f70c2aad640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f70bc13ee90 0x7f70bc1e6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33562/0 (socket says 172.21.15.64:33562) 2024-03-29T11:28:42.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.599+0000 7f70c2aad640 1 -- 172.21.15.64:0/2552682961 learned_addr learned my addr 172.21.15.64:0/2552682961 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.599+0000 7f70b2ffd640 10 client.?.objecter ms_handle_connect 0x7f70bc13ee90 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 client.?.objecter ms_handle_connect 0x7f70bc0af900 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 client.?.objecter ms_handle_connect 0x7f70bc0a9ac0 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397319881 0 0) 0x7f70bc071480 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a0004eb0 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552363597 0 0) 0x7f70bc071c40 con 0x7f70bc0af900 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.600+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70bc071480 con 0x7f70bc0af900 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185055083 0 0) 0x7f70bc0718a0 con 0x7f70bc13ee90 2024-03-29T11:28:42.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70bc071c40 con 0x7f70bc13ee90 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1705364596 0 0) 0x7f70a0004eb0 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70bc0718a0 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f70b8001bc0 con 0x7f70bc0a9ac0 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.597 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 723530062 0 0) 0x7f70bc071c40 con 0x7f70bc13ee90 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a0004eb0 con 0x7f70bc13ee90 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f70b4001cb0 con 0x7f70bc13ee90 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:42.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.601+0000 7f70b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2069203035 0 0) 0x7f70a0004eb0 con 0x7f70bc13ee90 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 >> v1:172.21.15.136:6789/0 conn(0x7f70bc0a9ac0 legacy=0x7f70bc0720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 >> v1:172.21.15.64:6789/0 conn(0x7f70bc0af900 legacy=0x7f70bc06f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 monclient: found mon.c 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc070040 con 0x7f70bc13ee90 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 10 monclient: _renew_subs 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc06ee20 con 0x7f70bc13ee90 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b40020c0 con 0x7f70bc13ee90 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:42.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f70b4004d40 con 0x7f70bc13ee90 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b2ffd640 10 monclient: dump: 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70b3fff640 4 set_mon_vals no callback set 2024-03-29T11:28:42.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 10 monclient: _renew_subs 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc06f120 con 0x7f70bc13ee90 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.602+0000 7f70c4510640 1 librados: init done 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.604+0000 7f70c4510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.604+0000 7f70c4510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.604+0000 7f70c4510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:42.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.604+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70bc075f10 con 0x7f70bc13ee90 2024-03-29T11:28:42.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.606+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f70b401ad10 con 0x7f70bc13ee90 2024-03-29T11:28:42.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f70b4084cf0 con 0x7f70bc13ee90 2024-03-29T11:28:42.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70b2ffd640 10 client.4415.objecter ms_dispatch 0x7f70bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:42.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70c22ac640 10 monclient: get_auth_request con 0x7f70a006dfe0 auth_method 0 2024-03-29T11:28:42.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70b2ffd640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:42.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70b2ffd640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:42.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70b2ffd640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:28:42.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.607+0000 7f70b2ffd640 10 client.4415.objecter ms_handle_connect 0x7f70a006dfe0 2024-03-29T11:28:42.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.616+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70b4050400 con 0x7f70bc13ee90 2024-03-29T11:28:42.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.616+0000 7f70b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.616+0000 7f70b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:42.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70c4510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70c4510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70c4510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:42.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70bc0701d0 con 0x7f70bc13ee90 2024-03-29T11:28:42.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70b2ffd640 1 -- 172.21.15.64:0/2552682961 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f70b4052500 con 0x7f70bc13ee90 2024-03-29T11:28:42.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.961+0000 7f70b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:42.953 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:42.953 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:42.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 10 librados: watch_flush enter 2024-03-29T11:28:42.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 10 librados: watch_flush exit 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 >> v1:172.21.15.136:6816/2312274743 conn(0x7f70a006dfe0 legacy=0x7f70a00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 10 monclient: shutdown 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.965+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 >> v1:172.21.15.64:6790/0 conn(0x7f70bc13ee90 legacy=0x7f70bc1e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.967+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 shutdown_connections 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.967+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 >> 172.21.15.64:0/2552682961 conn(0x7f70bc0b35e0 msgr2=0x7f70bc0ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.967+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 shutdown_connections 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.968+0000 7f70c4510640 1 -- 172.21.15.64:0/2552682961 wait complete. 2024-03-29T11:28:42.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:42.968+0000 7f70c4510640 1 librados: shutdown 2024-03-29T11:28:42.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:49.782 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:49.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.938+0000 7fef66f13640 10 monclient: get_monmap_and_config 2024-03-29T11:28:49.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.938+0000 7fef66f13640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.938+0000 7fef66f13640 10 monclient: monmap: 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:49.939091+0000 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:49.939091+0000 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:49.929 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:49.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.939+0000 7fef66f13640 1 Processor -- start 2024-03-29T11:28:49.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.940+0000 7fef66f13640 1 -- start start 2024-03-29T11:28:49.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.940+0000 7fef66f13640 10 monclient: init 2024-03-29T11:28:49.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient(hunting): picked mon.noname-a con 0x7fef600b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient(hunting): picked mon.noname-b con 0x7fef6014a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient(hunting): picked mon.noname-c con 0x7fef600ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600aba70 con 0x7fef600b0770 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600aecb0 con 0x7fef600ac5a0 2024-03-29T11:28:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600af150 con 0x7fef6014a700 2024-03-29T11:28:49.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.941+0000 7fef66f13640 10 monclient(hunting): authenticate will time out at 1466.211755s 2024-03-29T11:28:49.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef64caf640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fef600b0770 0x7fef60149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60061/0 (socket says 172.21.15.64:60061) 2024-03-29T11:28:49.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef64caf640 1 -- 172.21.15.64:0/1119124384 learned_addr learned my addr 172.21.15.64:0/1119124384 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:49.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495514322 0 0) 0x7fef600af150 con 0x7fef6014a700 2024-03-29T11:28:49.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef48003520 con 0x7fef6014a700 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.944+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3641329052 0 0) 0x7fef48003520 con 0x7fef6014a700 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 >> v1:172.21.15.136:6789/0 conn(0x7fef600ac5a0 legacy=0x7fef60151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 >> v1:172.21.15.64:6789/0 conn(0x7fef600b0770 legacy=0x7fef60149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600ac9d0 con 0x7fef6014a700 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fef50001670 con 0x7fef6014a700 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: dump: 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.938 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef66f13640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef66f13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef50001940 con 0x7fef6014a700 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef66f13640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef66f13640 4 set_mon_vals no callback set 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 1 -- 172.21.15.64:0/1119124384 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fef50001cd0 con 0x7fef6014a700 2024-03-29T11:28:49.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.945+0000 7fef5f7fe640 10 monclient: dump: 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:49.940 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.947+0000 7fef66f13640 10 monclient: shutdown 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.947+0000 7fef66f13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.947+0000 7fef66f13640 1 -- 172.21.15.64:0/1119124384 >> v1:172.21.15.64:6790/0 conn(0x7fef6014a700 legacy=0x7fef600abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.949+0000 7fef66f13640 1 -- 172.21.15.64:0/1119124384 shutdown_connections 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.949+0000 7fef66f13640 1 -- 172.21.15.64:0/1119124384 >> 172.21.15.64:0/1119124384 conn(0x7fef6006b660 msgr2=0x7fef6006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.949+0000 7fef66f13640 1 -- 172.21.15.64:0/1119124384 shutdown_connections 2024-03-29T11:28:49.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.949+0000 7fef66f13640 1 -- 172.21.15.64:0/1119124384 wait complete. 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 10 monclient: monmap: 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:49.945 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 1 librados: starting msgr at 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 1 librados: starting objecter 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 1 Processor -- start 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 1 -- start start 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 1 librados: setting wanted keys 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 1 librados: calling monclient init 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.950+0000 7fef66f13640 10 monclient: init 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient(hunting): picked mon.noname-c con 0x7fef600ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient(hunting): picked mon.noname-a con 0x7fef600b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient(hunting): picked mon.noname-b con 0x7fef6014a700 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60125260 con 0x7fef600b0770 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef601df920 con 0x7fef6014a700 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef601265e0 con 0x7fef600ac5a0 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.951+0000 7fef66f13640 10 monclient(hunting): authenticate will time out at 1466.221494s 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef654b0640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fef6014a700 0x7fef601dbed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:33198/0 (socket says 172.21.15.64:33198) 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef654b0640 1 -- 172.21.15.64:0/461722216 learned_addr learned my addr 172.21.15.64:0/461722216 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 10 client.?.objecter ms_handle_connect 0x7fef6014a700 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567378178 0 0) 0x7fef601df920 con 0x7fef6014a700 2024-03-29T11:28:49.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:28:49.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.952+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef4c004eb0 con 0x7fef6014a700 2024-03-29T11:28:49.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839780466 0 0) 0x7fef4c004eb0 con 0x7fef6014a700 2024-03-29T11:28:49.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef601df920 con 0x7fef6014a700 2024-03-29T11:28:49.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fef58001c60 con 0x7fef6014a700 2024-03-29T11:28:49.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:49.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.955+0000 7fef5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.956+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4161413494 0 0) 0x7fef601df920 con 0x7fef6014a700 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.956+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 >> v1:172.21.15.64:6790/0 conn(0x7fef600ac5a0 legacy=0x7fef601441f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 >> v1:172.21.15.64:6789/0 conn(0x7fef600b0770 legacy=0x7fef601db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 1 monclient: found mon.b 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef60126770 con 0x7fef6014a700 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 10 monclient: _renew_subs 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef60144900 con 0x7fef6014a700 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 10 monclient: _renew_subs 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef60070790 con 0x7fef6014a700 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.957+0000 7fef66f13640 1 librados: init done 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.958+0000 7fef66f13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.958+0000 7fef66f13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.958+0000 7fef66f13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:49.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.958+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef60070ae0 con 0x7fef6014a700 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef580020b0 con 0x7fef6014a700 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fef58003c20 con 0x7fef6014a700 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5d7fa640 10 monclient: dump: 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.956 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:49.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.959+0000 7fef5e7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.963+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fef58003d80 con 0x7fef6014a700 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.964+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fef58084da0 con 0x7fef6014a700 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.964+0000 7fef5d7fa640 10 client.4423.objecter ms_dispatch 0x7fef60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.964+0000 7fef5d7fa640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.964+0000 7fef5d7fa640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.964+0000 7fef5d7fa640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:28:49.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.965+0000 7fef64caf640 10 monclient: get_auth_request con 0x7fef4c06dc40 auth_method 0 2024-03-29T11:28:49.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.966+0000 7fef5d7fa640 10 client.4423.objecter ms_handle_connect 0x7fef4c06dc40 2024-03-29T11:28:49.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.968+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef580504b0 con 0x7fef6014a700 2024-03-29T11:28:49.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.968+0000 7fef5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:49.968+0000 7fef5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:50.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.305+0000 7fef66f13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.305+0000 7fef66f13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.305+0000 7fef66f13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:28:50.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.305+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef60075f10 con 0x7fef6014a700 2024-03-29T11:28:50.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.306+0000 7fef5d7fa640 1 -- 172.21.15.64:0/461722216 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fef580525b0 con 0x7fef6014a700 2024-03-29T11:28:50.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.306+0000 7fef5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.306+0000 7fef5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:50.296 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:50.297 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 10 librados: watch_flush enter 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 10 librados: watch_flush exit 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 >> v1:172.21.15.136:6816/2312274743 conn(0x7fef4c06dc40 legacy=0x7fef4c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 10 monclient: shutdown 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 >> v1:172.21.15.136:6789/0 conn(0x7fef6014a700 legacy=0x7fef601dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 shutdown_connections 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 >> 172.21.15.64:0/461722216 conn(0x7fef6006b660 msgr2=0x7fef60149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 shutdown_connections 2024-03-29T11:28:50.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.310+0000 7fef66f13640 1 -- 172.21.15.64:0/461722216 wait complete. 2024-03-29T11:28:50.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:50.311+0000 7fef66f13640 1 librados: shutdown 2024-03-29T11:28:50.321 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:57.128 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.288+0000 7fcf81038640 10 monclient: get_monmap_and_config 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.288+0000 7fcf81038640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.288+0000 7fcf81038640 10 monclient: monmap: 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:28:57.289304+0000 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:28:57.289304+0000 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.279 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:57.280 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:57.280 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:28:57.280 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:28:57.280 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.290+0000 7fcf81038640 1 Processor -- start 2024-03-29T11:28:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.290+0000 7fcf81038640 1 -- start start 2024-03-29T11:28:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.290+0000 7fcf81038640 10 monclient: init 2024-03-29T11:28:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient(hunting): picked mon.noname-a con 0x7fcf74006e30 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient(hunting): picked mon.noname-b con 0x7fcf74035e00 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient(hunting): picked mon.noname-c con 0x7fcf740c9670 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf74035970 con 0x7fcf74006e30 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf74036280 con 0x7fcf740c9670 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf74035640 con 0x7fcf74035e00 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.291+0000 7fcf81038640 10 monclient(hunting): authenticate will time out at 1473.561174s 2024-03-29T11:28:57.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7b7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcf74035e00 0x7fcf740c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41615/0 (socket says 172.21.15.64:41615) 2024-03-29T11:28:57.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7b7fe640 1 -- 172.21.15.64:0/968874375 learned_addr learned my addr 172.21.15.64:0/968874375 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:57.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376171577 0 0) 0x7fcf74035640 con 0x7fcf74035e00 2024-03-29T11:28:57.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:28:57.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c003520 con 0x7fcf74035e00 2024-03-29T11:28:57.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.294+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1317508512 0 0) 0x7fcf6c003520 con 0x7fcf74035e00 2024-03-29T11:28:57.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.295+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 >> v1:172.21.15.136:6789/0 conn(0x7fcf740c9670 legacy=0x7fcf740d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 >> v1:172.21.15.64:6789/0 conn(0x7fcf74006e30 legacy=0x7fcf740cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 1 monclient: found mon.noname-b 2024-03-29T11:28:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:28:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74034e20 con 0x7fcf74035e00 2024-03-29T11:28:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf81038640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:28:57.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf81038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:57.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcf7c04f730 con 0x7fcf74035e00 2024-03-29T11:28:57.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf7affd640 10 monclient: dump: 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:57.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.296+0000 7fcf81038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.297+0000 7fcf7affd640 1 -- 172.21.15.64:0/968874375 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf7c075f10 con 0x7fcf74035e00 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.297+0000 7fcf7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.297+0000 7fcf81038640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.297+0000 7fcf81038640 4 set_mon_vals no callback set 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 10 monclient: shutdown 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 1 -- 172.21.15.64:0/968874375 >> v1:172.21.15.64:6790/0 conn(0x7fcf74035e00 legacy=0x7fcf740c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 1 -- 172.21.15.64:0/968874375 shutdown_connections 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 1 -- 172.21.15.64:0/968874375 >> 172.21.15.64:0/968874375 conn(0x7fcf74005c00 msgr2=0x7fcf74006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 1 -- 172.21.15.64:0/968874375 shutdown_connections 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.298+0000 7fcf81038640 1 -- 172.21.15.64:0/968874375 wait complete. 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.299+0000 7fcf81038640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.299+0000 7fcf81038640 10 monclient: monmap: 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:28:57.294 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.299+0000 7fcf81038640 1 librados: starting msgr at 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.299+0000 7fcf81038640 1 librados: starting objecter 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.300+0000 7fcf81038640 1 Processor -- start 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.300+0000 7fcf81038640 1 -- start start 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.300+0000 7fcf81038640 1 librados: setting wanted keys 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.300+0000 7fcf81038640 1 librados: calling monclient init 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.300+0000 7fcf81038640 10 monclient: init 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient(hunting): picked mon.noname-c con 0x7fcf74006e30 addr v1:172.21.15.64:6790/0 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient(hunting): picked mon.noname-b con 0x7fcf74035e00 addr v1:172.21.15.136:6789/0 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient(hunting): picked mon.noname-a con 0x7fcf740c9670 addr v1:172.21.15.64:6789/0 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf740333a0 con 0x7fcf740c9670 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf740c13f0 con 0x7fcf74035e00 2024-03-29T11:28:57.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf74033010 con 0x7fcf74006e30 2024-03-29T11:28:57.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.301+0000 7fcf81038640 10 monclient(hunting): authenticate will time out at 1473.571309s 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.305+0000 7fcf7bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcf74006e30 0x7fcf74160a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41673/0 (socket says 172.21.15.64:41673) 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.305+0000 7fcf7bfff640 1 -- 172.21.15.64:0/3768306625 learned_addr learned my addr 172.21.15.64:0/3768306625 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.305+0000 7fcf78ff9640 10 client.?.objecter ms_handle_connect 0x7fcf74006e30 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.305+0000 7fcf78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658527699 0 0) 0x7fcf74033010 con 0x7fcf74006e30 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf60004eb0 con 0x7fcf74006e30 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 client.?.objecter ms_handle_connect 0x7fcf74035e00 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290698342 0 0) 0x7fcf740c13f0 con 0x7fcf74035e00 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74033010 con 0x7fcf74035e00 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 client.?.objecter ms_handle_connect 0x7fcf740c9670 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.307+0000 7fcf78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182105240 0 0) 0x7fcf740333a0 con 0x7fcf740c9670 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf740c13f0 con 0x7fcf740c9670 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583901244 0 0) 0x7fcf60004eb0 con 0x7fcf74006e30 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf740333a0 con 0x7fcf74006e30 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcf68001930 con 0x7fcf74006e30 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.307 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310164577 0 0) 0x7fcf740c13f0 con 0x7fcf740c9670 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf60004eb0 con 0x7fcf740c9670 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcf70001710 con 0x7fcf740c9670 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.308+0000 7fcf78ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:57.308 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 750406166 0 0) 0x7fcf740333a0 con 0x7fcf74006e30 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 >> v1:172.21.15.136:6789/0 conn(0x7fcf74035e00 legacy=0x7fcf740c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 >> v1:172.21.15.64:6789/0 conn(0x7fcf740c9670 legacy=0x7fcf740c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 1 monclient: found mon.c 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:57.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7415e5a0 con 0x7fcf74006e30 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf68003610 con 0x7fcf74006e30 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.309+0000 7fcf7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 10 monclient: _renew_subs 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf74161170 con 0x7fcf74006e30 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 10 monclient: _renew_subs 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf74033a10 con 0x7fcf74006e30 2024-03-29T11:28:57.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf81038640 1 librados: init done 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fcf680039a0 con 0x7fcf74006e30 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.310+0000 7fcf78ff9640 10 monclient: dump: 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:28:57.311 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.312+0000 7fcf81038640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.312+0000 7fcf81038640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.312+0000 7fcf81038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.312+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf74007b70 con 0x7fcf74006e30 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.312+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fcf68003b00 con 0x7fcf74006e30 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.313+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fcf68084a50 con 0x7fcf74006e30 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.313+0000 7fcf78ff9640 10 client.4424.objecter ms_dispatch 0x7fcf7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.313+0000 7fcf78ff9640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.313+0000 7fcf78ff9640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.313+0000 7fcf78ff9640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:28:57.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.313+0000 7fcf7b7fe640 10 monclient: get_auth_request con 0x7fcf6006de20 auth_method 0 2024-03-29T11:28:57.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.316+0000 7fcf78ff9640 10 client.4424.objecter ms_handle_connect 0x7fcf6006de20 2024-03-29T11:28:57.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.322+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf68050330 con 0x7fcf74006e30 2024-03-29T11:28:57.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.322+0000 7fcf78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.322+0000 7fcf78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:57.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.653+0000 7fcf81038640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.653+0000 7fcf81038640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.653+0000 7fcf81038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:28:57.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.653+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf74005120 con 0x7fcf74006e30 2024-03-29T11:28:57.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.655+0000 7fcf78ff9640 1 -- 172.21.15.64:0/3768306625 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fcf68052260 con 0x7fcf74006e30 2024-03-29T11:28:57.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.655+0000 7fcf78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.655+0000 7fcf78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:57.647 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:28:57.647 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:57.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.661+0000 7fcf81038640 10 librados: watch_flush enter 2024-03-29T11:28:57.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.661+0000 7fcf81038640 10 librados: watch_flush exit 2024-03-29T11:28:57.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.661+0000 7fcf81038640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.661+0000 7fcf81038640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 >> v1:172.21.15.136:6816/2312274743 conn(0x7fcf6006de20 legacy=0x7fcf600702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 10 monclient: shutdown 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 >> v1:172.21.15.64:6790/0 conn(0x7fcf74006e30 legacy=0x7fcf74160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 shutdown_connections 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 >> 172.21.15.64:0/3768306625 conn(0x7fcf74005c00 msgr2=0x7fcf74005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 shutdown_connections 2024-03-29T11:28:57.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 -- 172.21.15.64:0/3768306625 wait complete. 2024-03-29T11:28:57.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:28:57.662+0000 7fcf81038640 1 librados: shutdown 2024-03-29T11:28:57.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:04.477 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:04.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.635+0000 7fd29bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.635+0000 7fd29bfff640 10 monclient: build_initial_monmap 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.635+0000 7fd29bfff640 10 monclient: monmap: 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:04.636820+0000 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:04.636820+0000 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:04.628 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:04.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.639+0000 7fd29bfff640 1 Processor -- start 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.640+0000 7fd29bfff640 1 -- start start 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.640+0000 7fd29bfff640 10 monclient: init 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.640+0000 7fd29bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.640+0000 7fd29bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fd294034850 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fd29402a640 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fd2940327d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29402c820 con 0x7fd294034850 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29402a120 con 0x7fd29402a640 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940c75b0 con 0x7fd2940327d0 2024-03-29T11:29:04.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:04.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.641+0000 7fd29bfff640 10 monclient(hunting): authenticate will time out at 1480.910953s 2024-03-29T11:29:04.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.646+0000 7fd29a7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd29402a640 0x7fd2940320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:39227/0 (socket says 172.21.15.64:39227) 2024-03-29T11:29:04.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.646+0000 7fd29a7fc640 1 -- 172.21.15.64:0/2210890100 learned_addr learned my addr 172.21.15.64:0/2210890100 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237621296 0 0) 0x7fd29402a120 con 0x7fd29402a640 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284003520 con 0x7fd29402a640 2024-03-29T11:29:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328896841 0 0) 0x7fd284003520 con 0x7fd29402a640 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 >> v1:172.21.15.64:6790/0 conn(0x7fd2940327d0 legacy=0x7fd2940cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.647+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 >> v1:172.21.15.64:6789/0 conn(0x7fd294034850 legacy=0x7fd2940c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 1 monclient: found mon.noname-c 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294033110 con 0x7fd29402a640 2024-03-29T11:29:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd29bfff640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd29bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd29c04f730 con 0x7fd29402a640 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: dump: 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.640 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd29bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd29c0008d0 con 0x7fd29402a640 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 1 -- 172.21.15.64:0/2210890100 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd29c0758b0 con 0x7fd29402a640 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd299ffb640 10 monclient: dump: 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.641 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.642 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.642 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:04.642 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:04.642 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:04.642 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:04.642 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:04.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd29bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:04.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.648+0000 7fd29bfff640 4 set_mon_vals no callback set 2024-03-29T11:29:04.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.649+0000 7fd29bfff640 10 monclient: shutdown 2024-03-29T11:29:04.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.649+0000 7fd29bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:04.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.649+0000 7fd29bfff640 1 -- 172.21.15.64:0/2210890100 >> v1:172.21.15.136:6789/0 conn(0x7fd29402a640 legacy=0x7fd2940320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.650+0000 7fd29bfff640 1 -- 172.21.15.64:0/2210890100 shutdown_connections 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.650+0000 7fd29bfff640 1 -- 172.21.15.64:0/2210890100 >> 172.21.15.64:0/2210890100 conn(0x7fd2940058f0 msgr2=0x7fd294005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.650+0000 7fd29bfff640 1 -- 172.21.15.64:0/2210890100 shutdown_connections 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.650+0000 7fd29bfff640 1 -- 172.21.15.64:0/2210890100 wait complete. 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.651+0000 7fd29bfff640 10 monclient: build_initial_monmap 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.651+0000 7fd29bfff640 10 monclient: monmap: 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:04.644 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.651+0000 7fd29bfff640 1 librados: starting msgr at 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.651+0000 7fd29bfff640 1 librados: starting objecter 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.651+0000 7fd29bfff640 1 Processor -- start 2024-03-29T11:29:04.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.654+0000 7fd29bfff640 1 -- start start 2024-03-29T11:29:04.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.654+0000 7fd29bfff640 1 librados: setting wanted keys 2024-03-29T11:29:04.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.654+0000 7fd29bfff640 1 librados: calling monclient init 2024-03-29T11:29:04.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.654+0000 7fd29bfff640 10 monclient: init 2024-03-29T11:29:04.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:04.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:04.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fd29402a640 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:04.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fd2940327d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:04.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fd294034850 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:04.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd294169840 con 0x7fd294034850 2024-03-29T11:29:04.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940c3240 con 0x7fd29402a640 2024-03-29T11:29:04.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29416a3e0 con 0x7fd2940327d0 2024-03-29T11:29:04.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:04.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.655+0000 7fd29bfff640 10 monclient(hunting): authenticate will time out at 1480.925143s 2024-03-29T11:29:04.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.658+0000 7fd29affd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd29402a640 0x7fd2940c0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:39327/0 (socket says 172.21.15.64:39327) 2024-03-29T11:29:04.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.658+0000 7fd29affd640 1 -- 172.21.15.64:0/1749506186 learned_addr learned my addr 172.21.15.64:0/1749506186 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:04.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 10 client.?.objecter ms_handle_connect 0x7fd29402a640 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309671398 0 0) 0x7fd2940c3240 con 0x7fd29402a640 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:29:04.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.659+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd288004eb0 con 0x7fd29402a640 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 client.?.objecter ms_handle_connect 0x7fd2940327d0 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 client.?.objecter ms_handle_connect 0x7fd294034850 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364627863 0 0) 0x7fd29416a3e0 con 0x7fd2940327d0 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2940c3240 con 0x7fd2940327d0 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147813640 0 0) 0x7fd294169840 con 0x7fd294034850 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29416a3e0 con 0x7fd294034850 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884084429 0 0) 0x7fd2940c3240 con 0x7fd2940327d0 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd294169840 con 0x7fd2940327d0 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd29c04f400 con 0x7fd2940327d0 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): dump: 2024-03-29T11:29:04.659 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.663+0000 7fd283fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3391580066 0 0) 0x7fd294169840 con 0x7fd2940327d0 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 >> v1:172.21.15.136:6789/0 conn(0x7fd29402a640 legacy=0x7fd2940c0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 >> v1:172.21.15.64:6789/0 conn(0x7fd294034850 legacy=0x7fd2941741c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:04.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 1 monclient: found mon.c 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29416a570 con 0x7fd2940327d0 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd283fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 10 monclient: _renew_subs 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2940c3940 con 0x7fd2940327d0 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 10 monclient: _renew_subs 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:04.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2940c1900 con 0x7fd2940327d0 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.664+0000 7fd29bfff640 1 librados: init done 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd281ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd281ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd281ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd281ffb640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c005080 con 0x7fd2940327d0 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd29c04fdd0 con 0x7fd2940327d0 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd283fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd29c04f400 con 0x7fd2940327d0 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd283fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.665+0000 7fd283fff640 10 monclient: dump: 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.662 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:04.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.666+0000 7fd2997fa640 4 set_mon_vals no callback set 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.668+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fd29c067bf0 con 0x7fd2940327d0 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.669+0000 7fd29affd640 10 monclient: get_auth_request con 0x7fd28806dd90 auth_method 0 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.669+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fd29c0ff0f0 con 0x7fd2940327d0 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.669+0000 7fd283fff640 10 client.4427.objecter ms_dispatch 0x7fd29c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.669+0000 7fd283fff640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.669+0000 7fd283fff640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.669+0000 7fd283fff640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:29:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.672+0000 7fd283fff640 10 client.4427.objecter ms_handle_connect 0x7fd28806dd90 2024-03-29T11:29:04.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.677+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd29c0caa30 con 0x7fd2940327d0 2024-03-29T11:29:04.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.677+0000 7fd283fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:04.677+0000 7fd283fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:04.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.006+0000 7fd281ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.006+0000 7fd281ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.006+0000 7fd281ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:04.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.006+0000 7fd281ffb640 1 -- 172.21.15.64:0/1749506186 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd27c005330 con 0x7fd2940327d0 2024-03-29T11:29:04.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.007+0000 7fd283fff640 1 -- 172.21.15.64:0/1749506186 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fd29c0cc960 con 0x7fd2940327d0 2024-03-29T11:29:04.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.007+0000 7fd283fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.007+0000 7fd283fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:04.999 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:04.999 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:05.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 10 librados: watch_flush enter 2024-03-29T11:29:05.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 10 librados: watch_flush exit 2024-03-29T11:29:05.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:29:05.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:29:05.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 >> v1:172.21.15.136:6816/2312274743 conn(0x7fd28806dd90 legacy=0x7fd288070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 10 monclient: shutdown 2024-03-29T11:29:05.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:05.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.011+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 >> v1:172.21.15.64:6790/0 conn(0x7fd2940327d0 legacy=0x7fd294169cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.012+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 shutdown_connections 2024-03-29T11:29:05.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.012+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 >> 172.21.15.64:0/1749506186 conn(0x7fd2940058f0 msgr2=0x7fd294036a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:05.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.012+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 shutdown_connections 2024-03-29T11:29:05.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.012+0000 7fd29bfff640 1 -- 172.21.15.64:0/1749506186 wait complete. 2024-03-29T11:29:05.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:05.012+0000 7fd29bfff640 1 librados: shutdown 2024-03-29T11:29:05.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:11.828 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.991+0000 7fe9ce699640 10 monclient: get_monmap_and_config 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.991+0000 7fe9ce699640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.991+0000 7fe9ce699640 10 monclient: monmap: 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:11.992663+0000 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:11.992663+0000 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:11.983 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:11.984 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:11.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.994+0000 7fe9ce699640 1 Processor -- start 2024-03-29T11:29:11.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.994+0000 7fe9ce699640 1 -- start start 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient: init 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c0006890 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c0006c70 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c002b4c0 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c002c830 con 0x7fe9c0006890 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c002afa0 con 0x7fe9c0006c70 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c002aba0 con 0x7fe9c002b4c0 2024-03-29T11:29:11.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.995+0000 7fe9ce699640 10 monclient(hunting): authenticate will time out at 1488.265765s 2024-03-29T11:29:11.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9cd697640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe9c0006890 0x7fe9c00c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52861/0 (socket says 172.21.15.64:52861) 2024-03-29T11:29:11.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9cd697640 1 -- 172.21.15.64:0/1608403462 learned_addr learned my addr 172.21.15.64:0/1608403462 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:11.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938370449 0 0) 0x7fe9c002c830 con 0x7fe9c0006890 2024-03-29T11:29:11.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9bffff640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:11.999+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b4003520 con 0x7fe9c0006890 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.000+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632158604 0 0) 0x7fe9b4003520 con 0x7fe9c0006890 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.000+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 >> v1:172.21.15.64:6790/0 conn(0x7fe9c002b4c0 legacy=0x7fe9c00cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.000+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 >> v1:172.21.15.136:6789/0 conn(0x7fe9c0006c70 legacy=0x7fe9c00c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 1 monclient: found mon.noname-a 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c002ad30 con 0x7fe9c0006890 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe9b8001690 con 0x7fe9c0006890 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: dump: 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:11.997 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9ce699640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9ce699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b80017f0 con 0x7fe9c0006890 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 1 -- 172.21.15.64:0/1608403462 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe9b8001c40 con 0x7fe9c0006890 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9bffff640 10 monclient: dump: 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:11.998 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9ce699640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.001+0000 7fe9ce699640 4 set_mon_vals no callback set 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.002+0000 7fe9ce699640 10 monclient: shutdown 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.002+0000 7fe9ce699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.002+0000 7fe9ce699640 1 -- 172.21.15.64:0/1608403462 >> v1:172.21.15.64:6789/0 conn(0x7fe9c0006890 legacy=0x7fe9c00c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.004+0000 7fe9ce699640 1 -- 172.21.15.64:0/1608403462 shutdown_connections 2024-03-29T11:29:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.004+0000 7fe9ce699640 1 -- 172.21.15.64:0/1608403462 >> 172.21.15.64:0/1608403462 conn(0x7fe9c00058f0 msgr2=0x7fe9c0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:12.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.005+0000 7fe9ce699640 1 -- 172.21.15.64:0/1608403462 shutdown_connections 2024-03-29T11:29:12.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.009+0000 7fe9ce699640 1 -- 172.21.15.64:0/1608403462 wait complete. 2024-03-29T11:29:12.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.009+0000 7fe9ce699640 10 monclient: build_initial_monmap 2024-03-29T11:29:12.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.009+0000 7fe9ce699640 10 monclient: monmap: 2024-03-29T11:29:12.003 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:12.003 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:12.003 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:12.003 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:12.004 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:12.004 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:12.004 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:12.004 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:12.004 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:12.004 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:12.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 1 librados: starting msgr at 2024-03-29T11:29:12.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 1 librados: starting objecter 2024-03-29T11:29:12.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 1 Processor -- start 2024-03-29T11:29:12.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 1 -- start start 2024-03-29T11:29:12.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 1 librados: setting wanted keys 2024-03-29T11:29:12.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 1 librados: calling monclient init 2024-03-29T11:29:12.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.010+0000 7fe9ce699640 10 monclient: init 2024-03-29T11:29:12.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:12.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:12.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c0006890 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:12.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c002b4c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:12.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c00315b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:12.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0030110 con 0x7fe9c00315b0 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00ba890 con 0x7fe9c002b4c0 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0031be0 con 0x7fe9c0006890 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.011+0000 7fe9ce699640 10 monclient(hunting): authenticate will time out at 1488.281769s 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9cde98640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe9c00315b0 0x7fe9c016bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:53093/0 (socket says 172.21.15.64:53093) 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9cd697640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe9c0006890 0x7fe9c002e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45175/0 (socket says 172.21.15.64:45175) 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9cd697640 1 -- 172.21.15.64:0/2208417261 learned_addr learned my addr 172.21.15.64:0/2208417261 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9bdffb640 10 client.?.objecter ms_handle_connect 0x7fe9c0006890 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9bdffb640 10 client.?.objecter ms_handle_connect 0x7fe9c00315b0 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.015+0000 7fe9bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323711168 0 0) 0x7fe9c0031be0 con 0x7fe9c0006890 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:12.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac004eb0 con 0x7fe9c0006890 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825574454 0 0) 0x7fe9c0030110 con 0x7fe9c00315b0 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.016+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0031be0 con 0x7fe9c00315b0 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770424177 0 0) 0x7fe9ac004eb0 con 0x7fe9c0006890 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c0030110 con 0x7fe9c0006890 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1344412080 0 0) 0x7fe9c0031be0 con 0x7fe9c00315b0 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9ac004eb0 con 0x7fe9c00315b0 2024-03-29T11:29:12.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe9b8001b80 con 0x7fe9c0006890 2024-03-29T11:29:12.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:12.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): dump: 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:12.025 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe9c4001830 con 0x7fe9c00315b0 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient(hunting): dump: 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1308699464 0 0) 0x7fe9c0030110 con 0x7fe9c0006890 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 >> v1:172.21.15.136:6789/0 conn(0x7fe9c002b4c0 legacy=0x7fe9c00d2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 >> v1:172.21.15.64:6789/0 conn(0x7fe9c00315b0 legacy=0x7fe9c016bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 monclient: found mon.c 2024-03-29T11:29:12.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0031d70 con 0x7fe9c0006890 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.017+0000 7fe9bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 10 monclient: _renew_subs 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c00d36b0 con 0x7fe9c0006890 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 10 monclient: _renew_subs 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:12.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c016d340 con 0x7fe9c0006890 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.018+0000 7fe9ce699640 1 librados: init done 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b8001ce0 con 0x7fe9c0006890 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe9b8004cf0 con 0x7fe9c0006890 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9bdffb640 10 monclient: dump: 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9ce699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9ce699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9ce699640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9beffd640 4 set_mon_vals no callback set 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.019+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0004cd0 con 0x7fe9c0006890 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.024+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fe9b801acc0 con 0x7fe9c0006890 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.025+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fe9b8084b80 con 0x7fe9c0006890 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.025+0000 7fe9bdffb640 10 client.4430.objecter ms_dispatch 0x7fe9c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.025+0000 7fe9bdffb640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.025+0000 7fe9bdffb640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.025+0000 7fe9bdffb640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:29:12.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.029+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9b8050460 con 0x7fe9c0006890 2024-03-29T11:29:12.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.029+0000 7fe9cce96640 10 monclient: get_auth_request con 0x7fe9ac06df50 auth_method 0 2024-03-29T11:29:12.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.030+0000 7fe9bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:12.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.030+0000 7fe9bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:12.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.030+0000 7fe9bdffb640 10 client.4430.objecter ms_handle_connect 0x7fe9ac06df50 2024-03-29T11:29:12.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.365+0000 7fe9ce699640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.365+0000 7fe9ce699640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.365+0000 7fe9ce699640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:12.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.365+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c00d3840 con 0x7fe9c0006890 2024-03-29T11:29:12.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.366+0000 7fe9bdffb640 1 -- 172.21.15.64:0/2208417261 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fe9b8052390 con 0x7fe9c0006890 2024-03-29T11:29:12.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.366+0000 7fe9bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.366+0000 7fe9bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:12.360 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:12.360 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.370+0000 7fe9ce699640 10 librados: watch_flush enter 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.370+0000 7fe9ce699640 10 librados: watch_flush exit 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.370+0000 7fe9ce699640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.370+0000 7fe9ce699640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 >> v1:172.21.15.136:6816/2312274743 conn(0x7fe9ac06df50 legacy=0x7fe9ac070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 10 monclient: shutdown 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 >> v1:172.21.15.64:6790/0 conn(0x7fe9c0006890 legacy=0x7fe9c002e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 shutdown_connections 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 >> 172.21.15.64:0/2208417261 conn(0x7fe9c00058f0 msgr2=0x7fe9c00050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 shutdown_connections 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.371+0000 7fe9ce699640 1 -- 172.21.15.64:0/2208417261 wait complete. 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:12.372+0000 7fe9ce699640 1 librados: shutdown 2024-03-29T11:29:12.379 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:19.186 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:19.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.342+0000 7fb129bcf640 10 monclient: get_monmap_and_config 2024-03-29T11:29:19.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.342+0000 7fb129bcf640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.342+0000 7fb129bcf640 10 monclient: monmap: 2024-03-29T11:29:19.334 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:19.334 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:19.334 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:19.343427+0000 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:19.343427+0000 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:19.335 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:19.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.343+0000 7fb129bcf640 1 Processor -- start 2024-03-29T11:29:19.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.343+0000 7fb129bcf640 1 -- start start 2024-03-29T11:29:19.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.343+0000 7fb129bcf640 10 monclient: init 2024-03-29T11:29:19.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:19.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient(hunting): picked mon.noname-b con 0x7fb1241457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient(hunting): picked mon.noname-a con 0x7fb12413f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient(hunting): picked mon.noname-c con 0x7fb1240a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1240aaef0 con 0x7fb12413f650 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124145bd0 con 0x7fb1240a9d70 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124145eb0 con 0x7fb1241457f0 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.344+0000 7fb129bcf640 10 monclient(hunting): authenticate will time out at 1495.614669s 2024-03-29T11:29:19.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1237fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb1241457f0 0x7fb12413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50327/0 (socket says 172.21.15.64:50327) 2024-03-29T11:29:19.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1237fe640 1 -- 172.21.15.64:0/3971963990 learned_addr learned my addr 172.21.15.64:0/3971963990 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:19.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136388749 0 0) 0x7fb124145eb0 con 0x7fb1241457f0 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1227fc640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.349+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10c003520 con 0x7fb1241457f0 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218616753 0 0) 0x7fb10c003520 con 0x7fb1241457f0 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 >> v1:172.21.15.136:6789/0 conn(0x7fb1240a9d70 legacy=0x7fb12414d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 >> v1:172.21.15.64:6789/0 conn(0x7fb12413f650 legacy=0x7fb1240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 monclient: found mon.noname-b 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:29:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124146340 con 0x7fb1241457f0 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb114001690 con 0x7fb1241457f0 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: dump: 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:19.346 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb114001860 con 0x7fb1241457f0 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 1 -- 172.21.15.64:0/3971963990 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb114001cb0 con 0x7fb1241457f0 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.350+0000 7fb1227fc640 10 monclient: dump: 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.347 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.351+0000 7fb129bcf640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.351+0000 7fb129bcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.351+0000 7fb129bcf640 4 set_mon_vals no callback set 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.352+0000 7fb129bcf640 10 monclient: shutdown 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.352+0000 7fb129bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.352+0000 7fb129bcf640 1 -- 172.21.15.64:0/3971963990 >> v1:172.21.15.64:6790/0 conn(0x7fb1241457f0 legacy=0x7fb12413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.352+0000 7fb129bcf640 1 -- 172.21.15.64:0/3971963990 shutdown_connections 2024-03-29T11:29:19.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.352+0000 7fb129bcf640 1 -- 172.21.15.64:0/3971963990 >> 172.21.15.64:0/3971963990 conn(0x7fb124066a00 msgr2=0x7fb124066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.353+0000 7fb129bcf640 1 -- 172.21.15.64:0/3971963990 shutdown_connections 2024-03-29T11:29:19.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.354+0000 7fb129bcf640 1 -- 172.21.15.64:0/3971963990 wait complete. 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.355+0000 7fb129bcf640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.355+0000 7fb129bcf640 10 monclient: monmap: 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.356+0000 7fb129bcf640 1 librados: starting msgr at 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.356+0000 7fb129bcf640 1 librados: starting objecter 2024-03-29T11:29:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.356+0000 7fb129bcf640 1 Processor -- start 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.356+0000 7fb129bcf640 1 -- start start 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.357+0000 7fb129bcf640 1 librados: setting wanted keys 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.357+0000 7fb129bcf640 1 librados: calling monclient init 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.357+0000 7fb129bcf640 10 monclient: init 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.358+0000 7fb129bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.358+0000 7fb129bcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.358+0000 7fb129bcf640 10 monclient(hunting): picked mon.noname-b con 0x7fb1240a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.358+0000 7fb129bcf640 10 monclient(hunting): picked mon.noname-c con 0x7fb12413f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.358+0000 7fb129bcf640 10 monclient(hunting): picked mon.noname-a con 0x7fb1241457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.359+0000 7fb129bcf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12413a3d0 con 0x7fb1241457f0 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.359+0000 7fb129bcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124139de0 con 0x7fb1240a9d70 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.359+0000 7fb129bcf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124139f70 con 0x7fb12413f650 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.359+0000 7fb129bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.359+0000 7fb129bcf640 10 monclient(hunting): authenticate will time out at 1495.629131s 2024-03-29T11:29:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.361+0000 7fb1237fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb1240a9d70 0x7fb12413a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:53033/0 (socket says 172.21.15.64:53033) 2024-03-29T11:29:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.361+0000 7fb1237fe640 1 -- 172.21.15.64:0/834508823 learned_addr learned my addr 172.21.15.64:0/834508823 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.363+0000 7fb10bfff640 10 client.?.objecter ms_handle_connect 0x7fb1240a9d70 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15467023 0 0) 0x7fb124139de0 con 0x7fb1240a9d70 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb110004eb0 con 0x7fb1240a9d70 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918806806 0 0) 0x7fb110004eb0 con 0x7fb1240a9d70 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb124139de0 con 0x7fb1240a9d70 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.364+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb114001c30 con 0x7fb1240a9d70 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.365+0000 7fb10bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.365+0000 7fb10bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.365+0000 7fb10bfff640 10 monclient(hunting): dump: 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:19.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.365+0000 7fb10bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:19.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 10 client.?.objecter ms_handle_connect 0x7fb12413f650 2024-03-29T11:29:19.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2140446138 0 0) 0x7fb124139de0 con 0x7fb1240a9d70 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 >> v1:172.21.15.64:6790/0 conn(0x7fb12413f650 legacy=0x7fb1241da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 >> v1:172.21.15.64:6789/0 conn(0x7fb1241457f0 legacy=0x7fb12413c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 1 monclient: found mon.b 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124142380 con 0x7fb1240a9d70 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.366+0000 7fb10bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb10bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb10bfff640 10 client.?.objecter ms_handle_connect 0x7fb1241457f0 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb10bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 10 monclient: _renew_subs 2024-03-29T11:29:19.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:19.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1241425d0 con 0x7fb1240a9d70 2024-03-29T11:29:19.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:19.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 10 monclient: _renew_subs 2024-03-29T11:29:19.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:19.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb124135f20 con 0x7fb1240a9d70 2024-03-29T11:29:19.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.367+0000 7fb129bcf640 1 librados: init done 2024-03-29T11:29:19.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.370+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb114004a90 con 0x7fb1240a9d70 2024-03-29T11:29:19.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.370+0000 7fb10bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.370+0000 7fb109ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.370+0000 7fb121ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fb114004ee0 con 0x7fb1240a9d70 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb10bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb10bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb10bfff640 10 monclient: dump: 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:19.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb109ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb109ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.371+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb124075f10 con 0x7fb1240a9d70 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.372+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fb114005100 con 0x7fb1240a9d70 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.372+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fb114084c20 con 0x7fb1240a9d70 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.372+0000 7fb10bfff640 10 client.4435.objecter ms_dispatch 0x7fb124000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.372+0000 7fb10bfff640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.372+0000 7fb10bfff640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.373+0000 7fb10bfff640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.373+0000 7fb122ffd640 10 monclient: get_auth_request con 0x7fb11006db50 auth_method 0 2024-03-29T11:29:19.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.373+0000 7fb10bfff640 10 client.4435.objecter ms_handle_connect 0x7fb11006db50 2024-03-29T11:29:19.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.382+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb114050500 con 0x7fb1240a9d70 2024-03-29T11:29:19.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.383+0000 7fb10bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.383+0000 7fb10bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:19.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.725+0000 7fb109ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.725+0000 7fb109ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.725+0000 7fb109ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.725+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1240b3680 con 0x7fb1240a9d70 2024-03-29T11:29:19.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.727+0000 7fb10bfff640 1 -- 172.21.15.64:0/834508823 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fb114052430 con 0x7fb1240a9d70 2024-03-29T11:29:19.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.727+0000 7fb10bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.727+0000 7fb10bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:19.718 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:19.719 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.730+0000 7fb109ffb640 10 librados: watch_flush enter 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.730+0000 7fb109ffb640 10 librados: watch_flush exit 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.731+0000 7fb109ffb640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.731+0000 7fb109ffb640 10 client.4435.objecter successfully canceled tick 2024-03-29T11:29:19.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.733+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 >> v1:172.21.15.136:6816/2312274743 conn(0x7fb11006db50 legacy=0x7fb110070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.733+0000 7fb109ffb640 10 monclient: shutdown 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.733+0000 7fb109ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.733+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 >> v1:172.21.15.136:6789/0 conn(0x7fb1240a9d70 legacy=0x7fb12413a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.733+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 shutdown_connections 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.734+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 >> 172.21.15.64:0/834508823 conn(0x7fb124066a00 msgr2=0x7fb1240b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.734+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 shutdown_connections 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.734+0000 7fb109ffb640 1 -- 172.21.15.64:0/834508823 wait complete. 2024-03-29T11:29:19.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:19.734+0000 7fb109ffb640 1 librados: shutdown 2024-03-29T11:29:19.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:26.552 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.711+0000 7f2f256e9640 10 monclient: get_monmap_and_config 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.711+0000 7f2f256e9640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.711+0000 7f2f256e9640 10 monclient: monmap: 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:26.713713+0000 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:26.713713+0000 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:26.704 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:26.705 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:26.705 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:26.705 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:26.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.712+0000 7f2f256e9640 1 Processor -- start 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.712+0000 7f2f256e9640 1 -- start start 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.713+0000 7f2f256e9640 10 monclient: init 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.713+0000 7f2f256e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.713+0000 7f2f256e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.713+0000 7f2f256e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f2014a400 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f200aa500 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f20149800 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f200aca50 con 0x7f2f200aa500 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20149cb0 con 0x7f2f20149800 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f200acc80 con 0x7f2f2014a400 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f256e9640 10 monclient(hunting): authenticate will time out at 1502.984940s 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f24ee8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2f20149800 0x7f2f20151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:55806/0 (socket says 172.21.15.64:55806) 2024-03-29T11:29:26.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.714+0000 7f2f24ee8640 1 -- 172.21.15.64:0/2990453732 learned_addr learned my addr 172.21.15.64:0/2990453732 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:26.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.715+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389108790 0 0) 0x7f2f20149cb0 con 0x7f2f20149800 2024-03-29T11:29:26.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.715+0000 7f2f1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.715+0000 7f2f1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.715+0000 7f2f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.715+0000 7f2f1effd640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:29:26.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.715+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f08003520 con 0x7f2f20149800 2024-03-29T11:29:26.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.716+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3189049235 0 0) 0x7f2f08003520 con 0x7f2f20149800 2024-03-29T11:29:26.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.716+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 >> v1:172.21.15.64:6790/0 conn(0x7f2f2014a400 legacy=0x7f2f200aac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:26.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.716+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 >> v1:172.21.15.64:6789/0 conn(0x7f2f200aa500 legacy=0x7f2f200ad160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:26.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 1 monclient: found mon.noname-c 2024-03-29T11:29:26.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f200ac350 con 0x7f2f20149800 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2f18001a10 con 0x7f2f20149800 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.717+0000 7f2f1effd640 10 monclient: dump: 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.713 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f256e9640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f256e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f18001ca0 con 0x7f2f20149800 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f1effd640 1 -- 172.21.15.64:0/2990453732 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2f18002030 con 0x7f2f20149800 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f256e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f256e9640 4 set_mon_vals no callback set 2024-03-29T11:29:26.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.718+0000 7f2f1effd640 10 monclient: dump: 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.719+0000 7f2f256e9640 10 monclient: shutdown 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.719+0000 7f2f256e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.719+0000 7f2f256e9640 1 -- 172.21.15.64:0/2990453732 >> v1:172.21.15.136:6789/0 conn(0x7f2f20149800 legacy=0x7f2f20151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.719+0000 7f2f256e9640 1 -- 172.21.15.64:0/2990453732 shutdown_connections 2024-03-29T11:29:26.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.719+0000 7f2f256e9640 1 -- 172.21.15.64:0/2990453732 >> 172.21.15.64:0/2990453732 conn(0x7f2f200b3a30 msgr2=0x7f2f200b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.720+0000 7f2f256e9640 1 -- 172.21.15.64:0/2990453732 shutdown_connections 2024-03-29T11:29:26.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.720+0000 7f2f256e9640 1 -- 172.21.15.64:0/2990453732 wait complete. 2024-03-29T11:29:26.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.721+0000 7f2f256e9640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.721+0000 7f2f256e9640 10 monclient: monmap: 2024-03-29T11:29:26.722 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:26.722 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 1 librados: starting msgr at 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 1 librados: starting objecter 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 1 Processor -- start 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 1 -- start start 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 1 librados: setting wanted keys 2024-03-29T11:29:26.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 1 librados: calling monclient init 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.722+0000 7f2f256e9640 10 monclient: init 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.723+0000 7f2f256e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.723+0000 7f2f256e9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.723+0000 7f2f256e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f200aa500 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.724+0000 7f2f256e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f20149800 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.724+0000 7f2f1f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2f20149800 0x7f2f2013bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:57178/0 (socket says 172.21.15.64:57178) 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.724+0000 7f2f1f7fe640 1 -- 172.21.15.64:0/2407643412 learned_addr learned my addr 172.21.15.64:0/2407643412 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.724+0000 7f2f1cff9640 10 client.?.objecter ms_handle_connect 0x7f2f20149800 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.724+0000 7f2f1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f256e9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f2014a400 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20146690 con 0x7f2f200aa500 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 10 client.?.objecter ms_handle_connect 0x7f2f2014a400 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20147150 con 0x7f2f20149800 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2013ed50 con 0x7f2f2014a400 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f256e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f256e9640 10 monclient(hunting): authenticate will time out at 1502.996644s 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295414528 0 0) 0x7f2f20147150 con 0x7f2f20149800 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.725+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c004eb0 con 0x7f2f20149800 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 client.?.objecter ms_handle_connect 0x7f2f200aa500 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105981874 0 0) 0x7f2f2013ed50 con 0x7f2f2014a400 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f20147150 con 0x7f2f2014a400 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 298110301 0 0) 0x7f2f0c004eb0 con 0x7f2f20149800 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f2013ed50 con 0x7f2f20149800 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2f10001960 con 0x7f2f20149800 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): dump: 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.726 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.726+0000 7f2f1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095095477 0 0) 0x7f2f20146690 con 0x7f2f200aa500 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c004eb0 con 0x7f2f200aa500 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 44588691 0 0) 0x7f2f2013ed50 con 0x7f2f20149800 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 >> v1:172.21.15.64:6790/0 conn(0x7f2f2014a400 legacy=0x7f2f201e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 >> v1:172.21.15.64:6789/0 conn(0x7f2f200aa500 legacy=0x7f2f20145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 monclient: found mon.b 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f201435f0 con 0x7f2f20149800 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.727+0000 7f2f1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f100035f0 con 0x7f2f20149800 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 10 monclient: _renew_subs 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:26.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2013c410 con 0x7f2f20149800 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2f10003940 con 0x7f2f20149800 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f1cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f1cff9640 10 monclient: dump: 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:26.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:26.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 10 monclient: _renew_subs 2024-03-29T11:29:26.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:26.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f201e7210 con 0x7f2f20149800 2024-03-29T11:29:26.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.728+0000 7f2f256e9640 1 librados: init done 2024-03-29T11:29:26.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.730+0000 7f2f1dffb640 4 set_mon_vals no callback set 2024-03-29T11:29:26.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.730+0000 7f2f256e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.730+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f2f10003aa0 con 0x7f2f20149800 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.731+0000 7f2f256e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.731+0000 7f2f256e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.731+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2004f730 con 0x7f2f20149800 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.732+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f2f10084a10 con 0x7f2f20149800 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.732+0000 7f2f1cff9640 10 client.4441.objecter ms_dispatch 0x7f2f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.732+0000 7f2f1cff9640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.732+0000 7f2f1cff9640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.732+0000 7f2f1cff9640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.732+0000 7f2f1ffff640 10 monclient: get_auth_request con 0x7f2f0c06de40 auth_method 0 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.733+0000 7f2f1cff9640 10 client.4441.objecter ms_handle_connect 0x7f2f0c06de40 2024-03-29T11:29:26.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.738+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f100502f0 con 0x7f2f20149800 2024-03-29T11:29:26.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.738+0000 7f2f1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:26.738+0000 7f2f1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:27.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.089+0000 7f2f256e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:27.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.089+0000 7f2f256e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:27.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.089+0000 7f2f256e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:27.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.090+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f2013c5a0 con 0x7f2f20149800 2024-03-29T11:29:27.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.090+0000 7f2f1cff9640 1 -- 172.21.15.64:0/2407643412 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f2f10052220 con 0x7f2f20149800 2024-03-29T11:29:27.083 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:27.083 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:27.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.091+0000 7f2f1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:27.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.091+0000 7f2f1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:27.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.094+0000 7f2f256e9640 10 librados: watch_flush enter 2024-03-29T11:29:27.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 10 librados: watch_flush exit 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 10 client.4441.objecter successfully canceled tick 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 >> v1:172.21.15.136:6816/2312274743 conn(0x7f2f0c06de40 legacy=0x7f2f0c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 10 monclient: shutdown 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.095+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 >> v1:172.21.15.136:6789/0 conn(0x7f2f20149800 legacy=0x7f2f2013bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.096+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 shutdown_connections 2024-03-29T11:29:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.096+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 >> 172.21.15.64:0/2407643412 conn(0x7f2f200b3a30 msgr2=0x7f2f200b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.096+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 shutdown_connections 2024-03-29T11:29:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.096+0000 7f2f256e9640 1 -- 172.21.15.64:0/2407643412 wait complete. 2024-03-29T11:29:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:27.096+0000 7f2f256e9640 1 librados: shutdown 2024-03-29T11:29:27.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:33.914 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.092+0000 7f09aba63640 10 monclient: get_monmap_and_config 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.092+0000 7f09aba63640 10 monclient: build_initial_monmap 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.092+0000 7f09aba63640 10 monclient: monmap: 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:34.094143+0000 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:34.094143+0000 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.085 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:34.086 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:34.086 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:34.086 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.093+0000 7f09aba63640 1 Processor -- start 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.093+0000 7f09aba63640 1 -- start start 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.093+0000 7f09aba63640 10 monclient: init 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient(hunting): picked mon.noname-a con 0x7f09a40aa0e0 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient(hunting): picked mon.noname-b con 0x7f09a40adb40 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient(hunting): picked mon.noname-c con 0x7f09a4149eb0 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a414a320 con 0x7f09a40aa0e0 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a40ab840 con 0x7f09a4149eb0 2024-03-29T11:29:34.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a4072300 con 0x7f09a40adb40 2024-03-29T11:29:34.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:34.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.094+0000 7f09aba63640 10 monclient(hunting): authenticate will time out at 1510.365711s 2024-03-29T11:29:34.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f09aa801640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f09a4149eb0 0x7f09a4151b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:50590/0 (socket says 172.21.15.64:50590) 2024-03-29T11:29:34.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f09aa801640 1 -- 172.21.15.64:0/1460234356 learned_addr learned my addr 172.21.15.64:0/1460234356 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:34.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574081991 0 0) 0x7f09a40ab840 con 0x7f09a4149eb0 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f099bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f099bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f099bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f099bfff640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.095+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0990003520 con 0x7f09a4149eb0 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881613727 0 0) 0x7f09a414a320 con 0x7f09a40aa0e0 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a40ab840 con 0x7f09a40aa0e0 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702151548 0 0) 0x7f0990003520 con 0x7f09a4149eb0 2024-03-29T11:29:34.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 >> v1:172.21.15.64:6790/0 conn(0x7f09a40adb40 legacy=0x7f09a41497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 >> v1:172.21.15.64:6789/0 conn(0x7f09a40aa0e0 legacy=0x7f09a40ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 monclient: found mon.noname-c 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a40a9b30 con 0x7f09a4149eb0 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f09aba63640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f099c001a40 con 0x7f09a4149eb0 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f09aba63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:34.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f099bfff640 10 monclient: dump: 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.096+0000 7f09aba63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:34.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f099c001ba0 con 0x7f09a4149eb0 2024-03-29T11:29:34.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f099bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:34.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f099bfff640 1 -- 172.21.15.64:0/1460234356 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f099c001fb0 con 0x7f09a4149eb0 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f099bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f099bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f099bfff640 10 monclient: dump: 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f09aba63640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.097+0000 7f09aba63640 4 set_mon_vals no callback set 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 10 monclient: shutdown 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 1 -- 172.21.15.64:0/1460234356 >> v1:172.21.15.136:6789/0 conn(0x7f09a4149eb0 legacy=0x7f09a4151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 1 -- 172.21.15.64:0/1460234356 shutdown_connections 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 1 -- 172.21.15.64:0/1460234356 >> 172.21.15.64:0/1460234356 conn(0x7f09a40b36e0 msgr2=0x7f09a40b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 1 -- 172.21.15.64:0/1460234356 shutdown_connections 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.098+0000 7f09aba63640 1 -- 172.21.15.64:0/1460234356 wait complete. 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.099+0000 7f09aba63640 10 monclient: build_initial_monmap 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.099+0000 7f09aba63640 10 monclient: monmap: 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:34.094 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 1 librados: starting msgr at 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 1 librados: starting objecter 2024-03-29T11:29:34.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 1 Processor -- start 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 1 -- start start 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 1 librados: setting wanted keys 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 1 librados: calling monclient init 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.100+0000 7f09aba63640 10 monclient: init 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient(hunting): picked mon.noname-b con 0x7f09a40aa0e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:34.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient(hunting): picked mon.noname-c con 0x7f09a40adb40 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient(hunting): picked mon.noname-a con 0x7f09a4149eb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09a97ff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f09a40aa0e0 0x7f09a413f0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:50786/0 (socket says 172.21.15.64:50786) 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09a97ff640 1 -- 172.21.15.64:0/1672505115 learned_addr learned my addr 172.21.15.64:0/1672505115 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a41485e0 con 0x7f09a4149eb0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a4146e60 con 0x7f09a40aa0e0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a4147380 con 0x7f09a40adb40 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.101+0000 7f09aba63640 10 monclient(hunting): authenticate will time out at 1510.372864s 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 client.?.objecter ms_handle_connect 0x7f09a40aa0e0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 client.?.objecter ms_handle_connect 0x7f09a4149eb0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091979255 0 0) 0x7f09a4146e60 con 0x7f09a40aa0e0 2024-03-29T11:29:34.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0988004eb0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 client.?.objecter ms_handle_connect 0x7f09a40adb40 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.102+0000 7f0999ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198559265 0 0) 0x7f09a41485e0 con 0x7f09a4149eb0 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a4146e60 con 0x7f09a4149eb0 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848676015 0 0) 0x7f0988004eb0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09a41485e0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f09a0001a10 con 0x7f09a40aa0e0 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:34.101 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2068269082 0 0) 0x7f09a4146e60 con 0x7f09a4149eb0 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0988004eb0 con 0x7f09a4149eb0 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f099c001eb0 con 0x7f09a4149eb0 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3179138107 0 0) 0x7f09a41485e0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 >> v1:172.21.15.64:6790/0 conn(0x7f09a40adb40 legacy=0x7f09a41db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 >> v1:172.21.15.64:6789/0 conn(0x7f09a4149eb0 legacy=0x7f09a41e6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 monclient: found mon.b 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.103+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4147510 con 0x7f09a40aa0e0 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f0999ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f0999ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f0999ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 10 monclient: _renew_subs 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a4146940 con 0x7f09a40aa0e0 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 10 monclient: _renew_subs 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a413f9c0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.104+0000 7f09aba63640 1 librados: init done 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09a0001c50 con 0x7f09a40aa0e0 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f0999ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f09a0003a60 con 0x7f09a40aa0e0 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f0999ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f0999ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:34.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f0999ffb640 10 monclient: dump: 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f099b7fe640 4 set_mon_vals no callback set 2024-03-29T11:29:34.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.105+0000 7f09aba63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:34.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.106+0000 7f09aba63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:34.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.106+0000 7f09aba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.106+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a40aad00 con 0x7f09a40aa0e0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.107+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f09a0003bc0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f09a0084b60 con 0x7f09a40aa0e0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f0999ffb640 10 client.4447.objecter ms_dispatch 0x7f09a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f0999ffb640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f0999ffb640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f0999ffb640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f09a8ffe640 10 monclient: get_auth_request con 0x7f098806de90 auth_method 0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.108+0000 7f0999ffb640 10 client.4447.objecter ms_handle_connect 0x7f098806de90 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.114+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09a0050440 con 0x7f09a40aa0e0 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.114+0000 7f0999ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.114+0000 7f0999ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:34.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.511+0000 7f09aba63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.511+0000 7f09aba63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.511+0000 7f09aba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:34.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.511+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09a40008d0 con 0x7f09a40aa0e0 2024-03-29T11:29:34.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.513+0000 7f0999ffb640 1 -- 172.21.15.64:0/1672505115 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f09a0052370 con 0x7f09a40aa0e0 2024-03-29T11:29:34.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.513+0000 7f0999ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.513+0000 7f0999ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:34.505 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:34.505 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.516+0000 7f09aba63640 10 librados: watch_flush enter 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.516+0000 7f09aba63640 10 librados: watch_flush exit 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.516+0000 7f09aba63640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.516+0000 7f09aba63640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 >> v1:172.21.15.136:6816/2312274743 conn(0x7f098806de90 legacy=0x7f0988070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 10 monclient: shutdown 2024-03-29T11:29:34.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:34.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 >> v1:172.21.15.136:6789/0 conn(0x7f09a40aa0e0 legacy=0x7f09a413f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 shutdown_connections 2024-03-29T11:29:34.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 >> 172.21.15.64:0/1672505115 conn(0x7f09a40b36e0 msgr2=0x7f09a40adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:34.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 shutdown_connections 2024-03-29T11:29:34.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 -- 172.21.15.64:0/1672505115 wait complete. 2024-03-29T11:29:34.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:34.517+0000 7f09aba63640 1 librados: shutdown 2024-03-29T11:29:34.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:41.331 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:41.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.475+0000 7f36d944e640 10 monclient: get_monmap_and_config 2024-03-29T11:29:41.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.475+0000 7f36d944e640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.475+0000 7f36d944e640 10 monclient: monmap: 2024-03-29T11:29:41.467 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:41.476773+0000 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:41.476773+0000 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:41.468 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.476+0000 7f36d944e640 1 Processor -- start 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.476+0000 7f36d944e640 1 -- start start 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.476+0000 7f36d944e640 10 monclient: init 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient(hunting): picked mon.noname-c con 0x7f36d41457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:41.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient(hunting): picked mon.noname-b con 0x7f36d413f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient(hunting): picked mon.noname-a con 0x7f36d40a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40aaef0 con 0x7f36d40a9d70 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d4145bd0 con 0x7f36d41457f0 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d4145eb0 con 0x7f36d413f650 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.477+0000 7f36d944e640 10 monclient(hunting): authenticate will time out at 1517.747688s 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d2ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f36d41457f0 0x7f36d413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:59486/0 (socket says 172.21.15.64:59486) 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d2ffd640 1 -- 172.21.15.64:0/4278033086 learned_addr learned my addr 172.21.15.64:0/4278033086 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650419106 0 0) 0x7f36d4145eb0 con 0x7f36d413f650 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:29:41.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc003520 con 0x7f36d413f650 2024-03-29T11:29:41.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815174836 0 0) 0x7f36bc003520 con 0x7f36d413f650 2024-03-29T11:29:41.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 >> v1:172.21.15.136:6789/0 conn(0x7f36d41457f0 legacy=0x7f36d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.478+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 >> v1:172.21.15.64:6789/0 conn(0x7f36d40a9d70 legacy=0x7f36d414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:41.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.479+0000 7f36d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.479+0000 7f36d1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:41.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.479+0000 7f36d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.479+0000 7f36d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4146340 con 0x7f36d413f650 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f36c8001690 con 0x7f36d413f650 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: dump: 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.476 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.477 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.477 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:41.477 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:41.477 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:41.477 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d944e640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d944e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c8001920 con 0x7f36d413f650 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d944e640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d944e640 4 set_mon_vals no callback set 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 1 -- 172.21.15.64:0/4278033086 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f36c8001cb0 con 0x7f36d413f650 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.480+0000 7f36d1ffb640 10 monclient: dump: 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:41.478 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 10 monclient: shutdown 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 1 -- 172.21.15.64:0/4278033086 >> v1:172.21.15.64:6790/0 conn(0x7f36d413f650 legacy=0x7f36d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 1 -- 172.21.15.64:0/4278033086 shutdown_connections 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 1 -- 172.21.15.64:0/4278033086 >> 172.21.15.64:0/4278033086 conn(0x7f36d4066a00 msgr2=0x7f36d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 1 -- 172.21.15.64:0/4278033086 shutdown_connections 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.481+0000 7f36d944e640 1 -- 172.21.15.64:0/4278033086 wait complete. 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.482+0000 7f36d944e640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.482+0000 7f36d944e640 10 monclient: monmap: 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:41.480 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.482+0000 7f36d944e640 1 librados: starting msgr at 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.482+0000 7f36d944e640 1 librados: starting objecter 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.482+0000 7f36d944e640 1 Processor -- start 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 1 -- start start 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 1 librados: setting wanted keys 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 1 librados: calling monclient init 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient: init 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:41.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient(hunting): picked mon.noname-a con 0x7f36d40a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient(hunting): picked mon.noname-b con 0x7f36d413f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient(hunting): picked mon.noname-c con 0x7f36d41457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d41db5b0 con 0x7f36d40a9d70 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40ad7b0 con 0x7f36d413f650 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40ada90 con 0x7f36d41457f0 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.483+0000 7f36d944e640 10 monclient(hunting): authenticate will time out at 1517.753469s 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.484+0000 7f36d37fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f36d41457f0 0x7f36d40abd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:60478/0 (socket says 172.21.15.64:60478) 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.484+0000 7f36d37fe640 1 -- 172.21.15.64:0/40383515 learned_addr learned my addr 172.21.15.64:0/40383515 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.484+0000 7f36bb7fe640 10 client.?.objecter ms_handle_connect 0x7f36d41457f0 2024-03-29T11:29:41.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.484+0000 7f36bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.485+0000 7f36bb7fe640 10 client.?.objecter ms_handle_connect 0x7f36d413f650 2024-03-29T11:29:41.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.485+0000 7f36bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.485+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869055620 0 0) 0x7f36d40ada90 con 0x7f36d41457f0 2024-03-29T11:29:41.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:29:41.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c0004eb0 con 0x7f36d41457f0 2024-03-29T11:29:41.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657014127 0 0) 0x7f36d40ad7b0 con 0x7f36d413f650 2024-03-29T11:29:41.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:29:41.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d40ada90 con 0x7f36d413f650 2024-03-29T11:29:41.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1352915948 0 0) 0x7f36c0004eb0 con 0x7f36d41457f0 2024-03-29T11:29:41.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d40ad7b0 con 0x7f36d41457f0 2024-03-29T11:29:41.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f36cc001c60 con 0x7f36d41457f0 2024-03-29T11:29:41.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:41.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): dump: 2024-03-29T11:29:41.493 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:41.493 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:41.493 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.493 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.493 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.494 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.494 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:41.494 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:41.494 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271346128 0 0) 0x7f36d40ada90 con 0x7f36d413f650 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36c0004eb0 con 0x7f36d413f650 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f36c8001550 con 0x7f36d413f650 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 10 monclient(hunting): dump: 2024-03-29T11:29:41.495 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2698065359 0 0) 0x7f36d40ad7b0 con 0x7f36d41457f0 2024-03-29T11:29:41.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 >> v1:172.21.15.136:6789/0 conn(0x7f36d413f650 legacy=0x7f36d40ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.486+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 >> v1:172.21.15.64:6789/0 conn(0x7f36d40a9d70 legacy=0x7f36d4140440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 1 monclient: found mon.c 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d41db090 con 0x7f36d41457f0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36cc001dc0 con 0x7f36d41457f0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 10 monclient: _renew_subs 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d4140b50 con 0x7f36d41457f0 2024-03-29T11:29:41.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 10 monclient: _renew_subs 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d41da990 con 0x7f36d41457f0 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.487+0000 7f36d944e640 1 librados: init done 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.488+0000 7f36bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.488+0000 7f36d17fa640 4 set_mon_vals no callback set 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.488+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f36cc003bd0 con 0x7f36d41457f0 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.488+0000 7f36bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.488+0000 7f36bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:41.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.488+0000 7f36bb7fe640 10 monclient: dump: 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.489+0000 7f36d944e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.489+0000 7f36d944e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.489+0000 7f36d944e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.489+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b4005080 con 0x7f36d41457f0 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.490+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f36cc003d30 con 0x7f36d41457f0 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.490+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f36cc084d10 con 0x7f36d41457f0 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.490+0000 7f36bb7fe640 10 client.4451.objecter ms_dispatch 0x7f36d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.490+0000 7f36bb7fe640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.490+0000 7f36bb7fe640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.490+0000 7f36bb7fe640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.491+0000 7f36d2ffd640 10 monclient: get_auth_request con 0x7f36c006ddf0 auth_method 0 2024-03-29T11:29:41.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.491+0000 7f36bb7fe640 10 client.4451.objecter ms_handle_connect 0x7f36c006ddf0 2024-03-29T11:29:41.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.499+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36cc050650 con 0x7f36d41457f0 2024-03-29T11:29:41.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.499+0000 7f36bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.499+0000 7f36bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:41.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.815+0000 7f36d944e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.815+0000 7f36d944e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.815+0000 7f36d944e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:29:41.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.815+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b4005330 con 0x7f36d41457f0 2024-03-29T11:29:41.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.816+0000 7f36bb7fe640 1 -- 172.21.15.64:0/40383515 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f36cc052580 con 0x7f36d41457f0 2024-03-29T11:29:41.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.816+0000 7f36bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.816+0000 7f36bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:41.809 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:41.809 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:41.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 10 librados: watch_flush enter 2024-03-29T11:29:41.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 10 librados: watch_flush exit 2024-03-29T11:29:41.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 >> v1:172.21.15.136:6816/2312274743 conn(0x7f36c006ddf0 legacy=0x7f36c00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 10 monclient: shutdown 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 >> v1:172.21.15.64:6790/0 conn(0x7f36d41457f0 legacy=0x7f36d40abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 shutdown_connections 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 >> 172.21.15.64:0/40383515 conn(0x7f36d4066a00 msgr2=0x7f36d40b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 shutdown_connections 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.819+0000 7f36d944e640 1 -- 172.21.15.64:0/40383515 wait complete. 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:41.820+0000 7f36d944e640 1 librados: shutdown 2024-03-29T11:29:41.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:48.638 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:48.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.804+0000 7f92b48d2640 10 monclient: get_monmap_and_config 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.804+0000 7f92b48d2640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.804+0000 7f92b48d2640 10 monclient: monmap: 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:48.805784+0000 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:48.805784+0000 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:48.797 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.806+0000 7f92b48d2640 1 Processor -- start 2024-03-29T11:29:48.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.806+0000 7f92b48d2640 1 -- start start 2024-03-29T11:29:48.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.806+0000 7f92b48d2640 10 monclient: init 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.807+0000 7f92b48d2640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.807+0000 7f92b48d2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.807+0000 7f92b48d2640 10 monclient(hunting): picked mon.noname-b con 0x7f92b00a9750 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.807+0000 7f92b48d2640 10 monclient(hunting): picked mon.noname-a con 0x7f92b00aad20 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.809+0000 7f92b48d2640 10 monclient(hunting): picked mon.noname-c con 0x7f92b00a9b30 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.809+0000 7f92b48d2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00af4d0 con 0x7f92b00aad20 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.809+0000 7f92b48d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00af7b0 con 0x7f92b00a9b30 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.809+0000 7f92b48d2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00acab0 con 0x7f92b00a9750 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.809+0000 7f92b48d2640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.809+0000 7f92b48d2640 10 monclient(hunting): authenticate will time out at 1525.079275s 2024-03-29T11:29:48.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.812+0000 7f92ae59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f92b00aad20 0x7f92b0141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:39263/0 (socket says 172.21.15.64:39263) 2024-03-29T11:29:48.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.812+0000 7f92ae59c640 1 -- 172.21.15.64:0/1970883865 learned_addr learned my addr 172.21.15.64:0/1970883865 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:48.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051868099 0 0) 0x7f92b00af4d0 con 0x7f92b00aad20 2024-03-29T11:29:48.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:29:48.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9298003520 con 0x7f92b00aad20 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478921978 0 0) 0x7f9298003520 con 0x7f92b00aad20 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 >> v1:172.21.15.64:6790/0 conn(0x7f92b00a9750 legacy=0x7f92b00ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 >> v1:172.21.15.136:6789/0 conn(0x7f92b00a9b30 legacy=0x7f92b0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 monclient: found mon.noname-a 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00aba30 con 0x7f92b00aad20 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f92a0001670 con 0x7f92b00aad20 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: dump: 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.808 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92b48d2640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92b48d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a00017d0 con 0x7f92b00aad20 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92b48d2640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92b48d2640 4 set_mon_vals no callback set 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 1 -- 172.21.15.64:0/1970883865 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f92a0001be0 con 0x7f92b00aad20 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.813+0000 7f92add9b640 10 monclient: dump: 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:48.809 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.814+0000 7f92b48d2640 10 monclient: shutdown 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.814+0000 7f92b48d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.814+0000 7f92b48d2640 1 -- 172.21.15.64:0/1970883865 >> v1:172.21.15.64:6789/0 conn(0x7f92b00aad20 legacy=0x7f92b0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.814+0000 7f92b48d2640 1 -- 172.21.15.64:0/1970883865 shutdown_connections 2024-03-29T11:29:48.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.814+0000 7f92b48d2640 1 -- 172.21.15.64:0/1970883865 >> 172.21.15.64:0/1970883865 conn(0x7f92b00673e0 msgr2=0x7f92b00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.818+0000 7f92b48d2640 1 -- 172.21.15.64:0/1970883865 shutdown_connections 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.818+0000 7f92b48d2640 1 -- 172.21.15.64:0/1970883865 wait complete. 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.819+0000 7f92b48d2640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.819+0000 7f92b48d2640 10 monclient: monmap: 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:48.812 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.819+0000 7f92b48d2640 1 librados: starting msgr at 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.819+0000 7f92b48d2640 1 librados: starting objecter 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.820+0000 7f92b48d2640 1 Processor -- start 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.820+0000 7f92b48d2640 1 -- start start 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.820+0000 7f92b48d2640 1 librados: setting wanted keys 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.820+0000 7f92b48d2640 1 librados: calling monclient init 2024-03-29T11:29:48.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.820+0000 7f92b48d2640 10 monclient: init 2024-03-29T11:29:48.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.821+0000 7f92b48d2640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.821+0000 7f92b48d2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:48.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.821+0000 7f92b48d2640 10 monclient(hunting): picked mon.noname-b con 0x7f92b00a9750 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:48.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 10 monclient(hunting): picked mon.noname-c con 0x7f92b00a9b30 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:48.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 10 monclient(hunting): picked mon.noname-a con 0x7f92b0137940 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:48.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0130c10 con 0x7f92b0137940 2024-03-29T11:29:48.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b012c540 con 0x7f92b00a9750 2024-03-29T11:29:48.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0131ab0 con 0x7f92b00a9b30 2024-03-29T11:29:48.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.824+0000 7f92b48d2640 10 monclient(hunting): authenticate will time out at 1525.094632s 2024-03-29T11:29:48.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.826+0000 7f92af59e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f92b0137940 0x7f92b012be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:39499/0 (socket says 172.21.15.64:39499) 2024-03-29T11:29:48.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.826+0000 7f92af59e640 1 -- 172.21.15.64:0/1048482137 learned_addr learned my addr 172.21.15.64:0/1048482137 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:48.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 client.?.objecter ms_handle_connect 0x7f92b0137940 2024-03-29T11:29:48.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 client.?.objecter ms_handle_connect 0x7f92b00a9b30 2024-03-29T11:29:48.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585790445 0 0) 0x7f92b0130c10 con 0x7f92b0137940 2024-03-29T11:29:48.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:29:48.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.827+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c004ee0 con 0x7f92b0137940 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 client.?.objecter ms_handle_connect 0x7f92b00a9750 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 139546350 0 0) 0x7f929c004ee0 con 0x7f92b0137940 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b0130c10 con 0x7f92b0137940 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f92a80042e0 con 0x7f92b0137940 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): dump: 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.827 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845479139 0 0) 0x7f92b012c540 con 0x7f92b00a9750 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.830+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c004ee0 con 0x7f92b00a9750 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657590073 0 0) 0x7f929c004ee0 con 0x7f92b00a9750 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f929c0059a0 con 0x7f92b00a9750 2024-03-29T11:29:48.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f92a4001870 con 0x7f92b00a9750 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): dump: 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417727596 0 0) 0x7f92b0131ab0 con 0x7f92b00a9b30 2024-03-29T11:29:48.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c004ee0 con 0x7f92b00a9b30 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1123957 0 0) 0x7f929c0059a0 con 0x7f92b00a9750 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 >> v1:172.21.15.64:6790/0 conn(0x7f92b00a9b30 legacy=0x7f92b012b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 >> v1:172.21.15.64:6789/0 conn(0x7f92b0137940 legacy=0x7f92b012be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 monclient: found mon.b 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b006e740 con 0x7f92b00a9750 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.831+0000 7f92977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 10 monclient: _renew_subs 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b006ea20 con 0x7f92b00a9750 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 10 monclient: _renew_subs 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b012ead0 con 0x7f92b00a9750 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92b48d2640 1 librados: init done 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a4003610 con 0x7f92b00a9750 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f92a4001d80 con 0x7f92b00a9750 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.832+0000 7f92977fe640 10 monclient: dump: 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.833+0000 7f92b48d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.834+0000 7f92b48d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.834+0000 7f92b48d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.834+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b004f730 con 0x7f92b00a9750 2024-03-29T11:29:48.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.834+0000 7f92acd99640 4 set_mon_vals no callback set 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.835+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f92a40039e0 con 0x7f92b00a9750 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.836+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f92a4083790 con 0x7f92b00a9750 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.836+0000 7f92977fe640 10 client.4453.objecter ms_dispatch 0x7f92b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.836+0000 7f92977fe640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.836+0000 7f92977fe640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.836+0000 7f92977fe640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:29:48.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.836+0000 7f92ae59c640 10 monclient: get_auth_request con 0x7f929c06e260 auth_method 0 2024-03-29T11:29:48.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.837+0000 7f92977fe640 10 client.4453.objecter ms_handle_connect 0x7f929c06e260 2024-03-29T11:29:48.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.845+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92a404f070 con 0x7f92b00a9750 2024-03-29T11:29:48.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.845+0000 7f92977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:48.845+0000 7f92977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:49.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.172+0000 7f92b48d2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:49.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.172+0000 7f92b48d2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:49.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.172+0000 7f92b48d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:49.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.172+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b0131050 con 0x7f92b00a9750 2024-03-29T11:29:49.167 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:49.168 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:49.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.175+0000 7f92977fe640 1 -- 172.21.15.64:0/1048482137 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f92a4050fa0 con 0x7f92b00a9750 2024-03-29T11:29:49.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.175+0000 7f92977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:49.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.175+0000 7f92977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:49.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.178+0000 7f92b48d2640 10 librados: watch_flush enter 2024-03-29T11:29:49.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.178+0000 7f92b48d2640 10 librados: watch_flush exit 2024-03-29T11:29:49.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.178+0000 7f92b48d2640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.178+0000 7f92b48d2640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.179+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 >> v1:172.21.15.136:6816/2312274743 conn(0x7f929c06e260 legacy=0x7f929c070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.179+0000 7f92b48d2640 10 monclient: shutdown 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.179+0000 7f92b48d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.179+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 >> v1:172.21.15.136:6789/0 conn(0x7f92b00a9750 legacy=0x7f92b01313a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.179+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 shutdown_connections 2024-03-29T11:29:49.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.179+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 >> 172.21.15.64:0/1048482137 conn(0x7f92b00673e0 msgr2=0x7f92b01417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:49.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.180+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 shutdown_connections 2024-03-29T11:29:49.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.180+0000 7f92b48d2640 1 -- 172.21.15.64:0/1048482137 wait complete. 2024-03-29T11:29:49.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:49.180+0000 7f92b48d2640 1 librados: shutdown 2024-03-29T11:29:49.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:55.998 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:56.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.153+0000 7ffa0e990640 10 monclient: get_monmap_and_config 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.153+0000 7ffa0e990640 10 monclient: build_initial_monmap 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.153+0000 7ffa0e990640 10 monclient: monmap: 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:29:56.154391+0000 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:29:56.154391+0000 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:29:56.147 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:56.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.156+0000 7ffa0e990640 1 Processor -- start 2024-03-29T11:29:56.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.157+0000 7ffa0e990640 1 -- start start 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.157+0000 7ffa0e990640 10 monclient: init 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient(hunting): picked mon.noname-a con 0x7ffa080b4100 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient(hunting): picked mon.noname-b con 0x7ffa0813ee90 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient(hunting): picked mon.noname-c con 0x7ffa080af900 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080afd10 con 0x7ffa080b4100 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080af6f0 con 0x7ffa080af900 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080aac80 con 0x7ffa0813ee90 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:56.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.158+0000 7ffa0e990640 10 monclient(hunting): authenticate will time out at 1532.428402s 2024-03-29T11:29:56.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.161+0000 7ffa0cf2d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffa080af900 0x7ffa0814be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:52688/0 (socket says 172.21.15.64:52688) 2024-03-29T11:29:56.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.161+0000 7ffa0cf2d640 1 -- 172.21.15.64:0/2834713833 learned_addr learned my addr 172.21.15.64:0/2834713833 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:56.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177820328 0 0) 0x7ffa080af6f0 con 0x7ffa080af900 2024-03-29T11:29:56.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f0003520 con 0x7ffa080af900 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805127897 0 0) 0x7ffa080afd10 con 0x7ffa080b4100 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:56.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.162+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa080af6f0 con 0x7ffa080b4100 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948353873 0 0) 0x7ff9f0003520 con 0x7ffa080af900 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 >> v1:172.21.15.64:6790/0 conn(0x7ffa0813ee90 legacy=0x7ffa0813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 >> v1:172.21.15.64:6789/0 conn(0x7ffa080b4100 legacy=0x7ffa080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 1 monclient: found mon.noname-c 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa080aaf60 con 0x7ffa080af900 2024-03-29T11:29:56.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.163+0000 7ffa06ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa06ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff9f8001690 con 0x7ffa080af900 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa06ffd640 10 monclient: dump: 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:56.162 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa0e990640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.164+0000 7ffa0e990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.165+0000 7ffa06ffd640 1 -- 172.21.15.64:0/2834713833 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9f80018b0 con 0x7ffa080af900 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.165+0000 7ffa06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:56.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.165+0000 7ffa0e990640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.165+0000 7ffa0e990640 4 set_mon_vals no callback set 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.166+0000 7ffa0e990640 10 monclient: shutdown 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.166+0000 7ffa0e990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.166+0000 7ffa0e990640 1 -- 172.21.15.64:0/2834713833 >> v1:172.21.15.136:6789/0 conn(0x7ffa080af900 legacy=0x7ffa0814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.166+0000 7ffa0e990640 1 -- 172.21.15.64:0/2834713833 shutdown_connections 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.166+0000 7ffa0e990640 1 -- 172.21.15.64:0/2834713833 >> 172.21.15.64:0/2834713833 conn(0x7ffa080b35e0 msgr2=0x7ffa080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.166+0000 7ffa0e990640 1 -- 172.21.15.64:0/2834713833 shutdown_connections 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.167+0000 7ffa0e990640 1 -- 172.21.15.64:0/2834713833 wait complete. 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.167+0000 7ffa0e990640 10 monclient: build_initial_monmap 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.168+0000 7ffa0e990640 10 monclient: monmap: 2024-03-29T11:29:56.164 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:29:56.165 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.168+0000 7ffa0e990640 1 librados: starting msgr at 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.168+0000 7ffa0e990640 1 librados: starting objecter 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.168+0000 7ffa0e990640 1 Processor -- start 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.171+0000 7ffa0e990640 1 -- start start 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.171+0000 7ffa0e990640 1 librados: setting wanted keys 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.171+0000 7ffa0e990640 1 librados: calling monclient init 2024-03-29T11:29:56.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.171+0000 7ffa0e990640 10 monclient: init 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient(hunting): picked mon.noname-a con 0x7ffa080af900 addr v1:172.21.15.64:6789/0 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient(hunting): picked mon.noname-b con 0x7ffa080b4100 addr v1:172.21.15.136:6789/0 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0813ee90 addr v1:172.21.15.64:6790/0 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa081407b0 con 0x7ffa080af900 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa08140a90 con 0x7ffa080b4100 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0813f5c0 con 0x7ffa0813ee90 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:56.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.172+0000 7ffa0e990640 10 monclient(hunting): authenticate will time out at 1532.442131s 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa077fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffa080b4100 0x7ffa08127660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:55132/0 (socket says 172.21.15.64:55132) 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa077fe640 1 -- 172.21.15.64:0/3443429846 learned_addr learned my addr 172.21.15.64:0/3443429846 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 10 client.?.objecter ms_handle_connect 0x7ffa080b4100 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329697158 0 0) 0x7ffa08140a90 con 0x7ffa080b4100 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.176+0000 7ffa04ff9640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f4004eb0 con 0x7ffa080b4100 2024-03-29T11:29:56.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1568776446 0 0) 0x7ff9f4004eb0 con 0x7ffa080b4100 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f4005300 con 0x7ffa080b4100 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ffa00001cb0 con 0x7ffa080b4100 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:56.174 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3520552282 0 0) 0x7ff9f4005300 con 0x7ffa080b4100 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.177+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 >> v1:172.21.15.64:6790/0 conn(0x7ffa0813ee90 legacy=0x7ffa081e28f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 >> v1:172.21.15.64:6789/0 conn(0x7ffa080af900 legacy=0x7ffa08140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 1 monclient: found mon.b 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa081415c0 con 0x7ffa080b4100 2024-03-29T11:29:56.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.178+0000 7ffa04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 10 monclient: _renew_subs 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa08141810 con 0x7ffa080b4100 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 10 monclient: _renew_subs 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0812da10 con 0x7ffa080b4100 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa0e990640 1 librados: init done 2024-03-29T11:29:56.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa00003970 con 0x7ffa080b4100 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa04ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ffa00001cb0 con 0x7ffa080b4100 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa04ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.179+0000 7ffa04ff9640 10 monclient: dump: 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:29:56.177 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:29:56.178 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:29:56.178 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:29:56.178 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:29:56.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.180+0000 7ffa067fc640 4 set_mon_vals no callback set 2024-03-29T11:29:56.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.180+0000 7ffa0e990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.181+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7ffa00003ae0 con 0x7ffa080b4100 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.181+0000 7ffa0e990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.181+0000 7ffa0e990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.181+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8005080 con 0x7ffa080b4100 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.182+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7ffa000547e0 con 0x7ffa080b4100 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.182+0000 7ffa04ff9640 10 client.4459.objecter ms_dispatch 0x7ffa08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.182+0000 7ffa04ff9640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.182+0000 7ffa04ff9640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.182+0000 7ffa04ff9640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:29:56.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.182+0000 7ffa07fff640 10 monclient: get_auth_request con 0x7ff9f406e1c0 auth_method 0 2024-03-29T11:29:56.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.183+0000 7ffa04ff9640 10 client.4459.objecter ms_handle_connect 0x7ff9f406e1c0 2024-03-29T11:29:56.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.190+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa00050a80 con 0x7ffa080b4100 2024-03-29T11:29:56.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.190+0000 7ffa04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:56.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.190+0000 7ffa04ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:56.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.520+0000 7ffa0e990640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.520+0000 7ffa0e990640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.520+0000 7ffa0e990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:29:56.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.520+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9e8005330 con 0x7ffa080b4100 2024-03-29T11:29:56.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.521+0000 7ffa04ff9640 1 -- 172.21.15.64:0/3443429846 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7ffa00050c20 con 0x7ffa080b4100 2024-03-29T11:29:56.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.521+0000 7ffa04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.521+0000 7ffa04ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:56.513 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:29:56.513 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 10 librados: watch_flush enter 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 10 librados: watch_flush exit 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 >> v1:172.21.15.136:6816/2312274743 conn(0x7ff9f406e1c0 legacy=0x7ff9f4070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 10 monclient: shutdown 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:56.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 >> v1:172.21.15.136:6789/0 conn(0x7ffa080b4100 legacy=0x7ffa08127660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 shutdown_connections 2024-03-29T11:29:56.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.523+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 >> 172.21.15.64:0/3443429846 conn(0x7ffa080b35e0 msgr2=0x7ffa08143140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:56.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.524+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 shutdown_connections 2024-03-29T11:29:56.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.524+0000 7ffa0e990640 1 -- 172.21.15.64:0/3443429846 wait complete. 2024-03-29T11:29:56.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:29:56.524+0000 7ffa0e990640 1 librados: shutdown 2024-03-29T11:29:56.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:03.338 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:03.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.494+0000 7f48644f1640 10 monclient: get_monmap_and_config 2024-03-29T11:30:03.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.494+0000 7f48644f1640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.494+0000 7f48644f1640 10 monclient: monmap: 2024-03-29T11:30:03.490 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:03.495021+0000 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:03.495021+0000 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:03.491 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.498+0000 7f48644f1640 1 Processor -- start 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.498+0000 7f48644f1640 1 -- start start 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.498+0000 7f48644f1640 10 monclient: init 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.499+0000 7f48644f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.499+0000 7f48644f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.499+0000 7f48644f1640 10 monclient(hunting): picked mon.noname-b con 0x7f485c1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:03.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.499+0000 7f48644f1640 10 monclient(hunting): picked mon.noname-c con 0x7f485c13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:03.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.499+0000 7f48644f1640 10 monclient(hunting): picked mon.noname-a con 0x7f485c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:03.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.501+0000 7f48644f1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0aaef0 con 0x7f485c0a9d70 2024-03-29T11:30:03.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.501+0000 7f48644f1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c145bd0 con 0x7f485c13f650 2024-03-29T11:30:03.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.501+0000 7f48644f1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c145eb0 con 0x7f485c1457f0 2024-03-29T11:30:03.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.501+0000 7f48644f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.501+0000 7f48644f1640 10 monclient(hunting): authenticate will time out at 1539.770954s 2024-03-29T11:30:03.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.503+0000 7f4861a8c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f485c13f650 0x7f485c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:49459/0 (socket says 172.21.15.64:49459) 2024-03-29T11:30:03.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.503+0000 7f4861a8c640 1 -- 172.21.15.64:0/3492671493 learned_addr learned my addr 172.21.15.64:0/3492671493 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:03.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.505+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428168231 0 0) 0x7f485c145bd0 con 0x7f485c13f650 2024-03-29T11:30:03.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.505+0000 7f486128b640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.505+0000 7f486128b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.505+0000 7f486128b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.505+0000 7f486128b640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:30:03.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.505+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003520 con 0x7f485c13f650 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526285928 0 0) 0x7f485c0aaef0 con 0x7f485c0a9d70 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c145bd0 con 0x7f485c0a9d70 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310577466 0 0) 0x7f485c145bd0 con 0x7f485c0a9d70 2024-03-29T11:30:03.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 >> v1:172.21.15.64:6790/0 conn(0x7f485c1457f0 legacy=0x7f485c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.506+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 >> v1:172.21.15.136:6789/0 conn(0x7f485c13f650 legacy=0x7f485c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 1 monclient: found mon.noname-a 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c146340 con 0x7f485c0a9d70 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4854001920 con 0x7f485c0a9d70 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.507+0000 7f486128b640 10 monclient: dump: 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:03.503 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f48644f1640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f48644f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4854001b40 con 0x7f485c0a9d70 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f486128b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f486128b640 1 -- 172.21.15.64:0/3492671493 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4854001e50 con 0x7f485c0a9d70 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f486128b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f486128b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f486128b640 10 monclient: dump: 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f48644f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.508+0000 7f48644f1640 4 set_mon_vals no callback set 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.510+0000 7f48644f1640 10 monclient: shutdown 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.510+0000 7f48644f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.510+0000 7f48644f1640 1 -- 172.21.15.64:0/3492671493 >> v1:172.21.15.64:6789/0 conn(0x7f485c0a9d70 legacy=0x7f485c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.510+0000 7f48644f1640 1 -- 172.21.15.64:0/3492671493 shutdown_connections 2024-03-29T11:30:03.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.510+0000 7f48644f1640 1 -- 172.21.15.64:0/3492671493 >> 172.21.15.64:0/3492671493 conn(0x7f485c066a00 msgr2=0x7f485c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.513+0000 7f48644f1640 1 -- 172.21.15.64:0/3492671493 shutdown_connections 2024-03-29T11:30:03.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.514+0000 7f48644f1640 1 -- 172.21.15.64:0/3492671493 wait complete. 2024-03-29T11:30:03.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.516+0000 7f48644f1640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.516+0000 7f48644f1640 10 monclient: monmap: 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:03.508 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:03.509 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:03.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 1 librados: starting msgr at 2024-03-29T11:30:03.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 1 librados: starting objecter 2024-03-29T11:30:03.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 1 Processor -- start 2024-03-29T11:30:03.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 1 -- start start 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 1 librados: setting wanted keys 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 1 librados: calling monclient init 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.517+0000 7f48644f1640 10 monclient: init 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient(hunting): picked mon.noname-c con 0x7f485c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient(hunting): picked mon.noname-b con 0x7f485c13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient(hunting): picked mon.noname-a con 0x7f485c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c13b5b0 con 0x7f485c1457f0 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c1404b0 con 0x7f485c13f650 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c141fd0 con 0x7f485c0a9d70 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.518+0000 7f48644f1640 10 monclient(hunting): authenticate will time out at 1539.788542s 2024-03-29T11:30:03.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.524+0000 7f4862a8e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f485c1457f0 0x7f485c1dbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35321/0 (socket says 172.21.15.64:35321) 2024-03-29T11:30:03.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.525+0000 7f4862a8e640 1 -- 172.21.15.64:0/3222160676 learned_addr learned my addr 172.21.15.64:0/3222160676 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:03.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.526+0000 7f484effd640 10 client.?.objecter ms_handle_connect 0x7f485c0a9d70 2024-03-29T11:30:03.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.526+0000 7f484effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.526+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023191949 0 0) 0x7f485c141fd0 con 0x7f485c0a9d70 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840004eb0 con 0x7f485c0a9d70 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 client.?.objecter ms_handle_connect 0x7f485c13f650 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 client.?.objecter ms_handle_connect 0x7f485c1457f0 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2862940202 0 0) 0x7f4840004eb0 con 0x7f485c0a9d70 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f485c141fd0 con 0x7f485c0a9d70 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4850001890 con 0x7f485c0a9d70 2024-03-29T11:30:03.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.527+0000 7f484effd640 10 monclient(hunting): dump: 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:03.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758457611 0 0) 0x7f485c1404b0 con 0x7f485c13f650 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840004eb0 con 0x7f485c13f650 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1029485671 0 0) 0x7f485c141fd0 con 0x7f485c0a9d70 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 >> v1:172.21.15.136:6789/0 conn(0x7f485c13f650 legacy=0x7f485c1db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 >> v1:172.21.15.64:6789/0 conn(0x7f485c1457f0 legacy=0x7f485c1dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 1 monclient: found mon.c 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.528+0000 7f484effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f484effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c1dc440 con 0x7f485c0a9d70 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f484effd640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f484effd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f484effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 10 monclient: _renew_subs 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c140db0 con 0x7f485c0a9d70 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:03.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 10 monclient: _renew_subs 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c142ce0 con 0x7f485c0a9d70 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f48644f1640 1 librados: init done 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.529+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4850003610 con 0x7f485c0a9d70 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.530+0000 7f484effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.530+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f48500039e0 con 0x7f485c0a9d70 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.530+0000 7f484effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.530+0000 7f484effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.530+0000 7f484effd640 10 monclient: dump: 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:03.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.530+0000 7f4860a8a640 4 set_mon_vals no callback set 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.532+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4850003b40 con 0x7f485c0a9d70 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f48644f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f48644f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f48644f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4844005080 con 0x7f485c0a9d70 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f48500837e0 con 0x7f485c0a9d70 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f484effd640 10 client.4460.objecter ms_dispatch 0x7f485c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f484effd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f484effd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:03.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.533+0000 7f484effd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T11:30:03.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.534+0000 7f4861a8c640 10 monclient: get_auth_request con 0x7f484006dd60 auth_method 0 2024-03-29T11:30:03.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.534+0000 7f484effd640 10 client.4460.objecter ms_handle_connect 0x7f484006dd60 2024-03-29T11:30:03.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.542+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f485004f0c0 con 0x7f485c0a9d70 2024-03-29T11:30:03.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.542+0000 7f484effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.542+0000 7f484effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:03.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.881+0000 7f48644f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.881+0000 7f48644f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.881+0000 7f48644f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:03.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.881+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4844005330 con 0x7f485c0a9d70 2024-03-29T11:30:03.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.883+0000 7f484effd640 1 -- 172.21.15.64:0/3222160676 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f4850050ff0 con 0x7f485c0a9d70 2024-03-29T11:30:03.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.883+0000 7f484effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.883+0000 7f484effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:03.878 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:03.878 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.888+0000 7f48644f1640 10 librados: watch_flush enter 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 10 librados: watch_flush exit 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 10 client.4460.objecter successfully canceled tick 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 >> v1:172.21.15.136:6816/2312274743 conn(0x7f484006dd60 legacy=0x7f4840070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 10 monclient: shutdown 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 >> v1:172.21.15.64:6790/0 conn(0x7f485c0a9d70 legacy=0x7f485c1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 shutdown_connections 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 >> 172.21.15.64:0/3222160676 conn(0x7f485c066a00 msgr2=0x7f485c0b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 shutdown_connections 2024-03-29T11:30:03.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.889+0000 7f48644f1640 1 -- 172.21.15.64:0/3222160676 wait complete. 2024-03-29T11:30:03.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:03.890+0000 7f48644f1640 1 librados: shutdown 2024-03-29T11:30:03.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:10.707 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.868+0000 7f4ddb98d640 10 monclient: get_monmap_and_config 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.868+0000 7f4ddb98d640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.868+0000 7f4ddb98d640 10 monclient: monmap: 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:10.869611+0000 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:10.869611+0000 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:10.862 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.869+0000 7f4ddb98d640 1 Processor -- start 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 1 -- start start 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient: init 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient(hunting): picked mon.noname-b con 0x7f4dd40ae250 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient(hunting): picked mon.noname-c con 0x7f4dd413f970 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient(hunting): picked mon.noname-a con 0x7f4dd4142800 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40afe00 con 0x7f4dd4142800 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd4142c10 con 0x7f4dd413f970 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40afb20 con 0x7f4dd40ae250 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.870+0000 7f4ddb98d640 10 monclient(hunting): authenticate will time out at 1547.140792s 2024-03-29T11:30:10.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dd9729640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4dd40ae250 0x7f4dd413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:55337/0 (socket says 172.21.15.64:55337) 2024-03-29T11:30:10.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dd9729640 1 -- 172.21.15.64:0/190737699 learned_addr learned my addr 172.21.15.64:0/190737699 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644729690 0 0) 0x7f4dd40afe00 con 0x7f4dd4142800 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dcbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.874+0000 7f4dcbfff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc0003520 con 0x7f4dd4142800 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3169606280 0 0) 0x7f4dc0003520 con 0x7f4dd4142800 2024-03-29T11:30:10.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 >> v1:172.21.15.64:6790/0 conn(0x7f4dd40ae250 legacy=0x7f4dd413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 >> v1:172.21.15.136:6789/0 conn(0x7f4dd413f970 legacy=0x7f4dd41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 monclient: found mon.noname-a 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40aab80 con 0x7f4dd4142800 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4dc4001690 con 0x7f4dd4142800 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.875+0000 7f4dcbfff640 10 monclient: dump: 2024-03-29T11:30:10.875 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4ddb98d640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4ddb98d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dc40017f0 con 0x7f4dd4142800 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4dcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4dcbfff640 1 -- 172.21.15.64:0/190737699 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4dc4001c40 con 0x7f4dd4142800 2024-03-29T11:30:10.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4dcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4dcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4dcbfff640 10 monclient: dump: 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4ddb98d640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.876+0000 7f4ddb98d640 4 set_mon_vals no callback set 2024-03-29T11:30:10.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 10 monclient: shutdown 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 1 -- 172.21.15.64:0/190737699 >> v1:172.21.15.64:6789/0 conn(0x7f4dd4142800 legacy=0x7f4dd414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 1 -- 172.21.15.64:0/190737699 shutdown_connections 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 1 -- 172.21.15.64:0/190737699 >> 172.21.15.64:0/190737699 conn(0x7f4dd40b35e0 msgr2=0x7f4dd40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 1 -- 172.21.15.64:0/190737699 shutdown_connections 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.877+0000 7f4ddb98d640 1 -- 172.21.15.64:0/190737699 wait complete. 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.878+0000 7f4ddb98d640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.878+0000 7f4ddb98d640 10 monclient: monmap: 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.878 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 1 librados: starting msgr at 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 1 librados: starting objecter 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 1 Processor -- start 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 1 -- start start 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 1 librados: setting wanted keys 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 1 librados: calling monclient init 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.879+0000 7f4ddb98d640 10 monclient: init 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.880+0000 7f4ddb98d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.880+0000 7f4ddb98d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:10.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.880+0000 7f4ddb98d640 10 monclient(hunting): picked mon.noname-c con 0x7f4dd40ae250 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.883+0000 7f4ddb98d640 10 monclient(hunting): picked mon.noname-b con 0x7f4dd413f970 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.883+0000 7f4ddb98d640 10 monclient(hunting): picked mon.noname-a con 0x7f4dd4142800 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4ddb98d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd41e47f0 con 0x7f4dd4142800 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4ddb98d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd41390b0 con 0x7f4dd413f970 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4ddb98d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd4143510 con 0x7f4dd40ae250 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4ddb98d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4ddb98d640 10 monclient(hunting): authenticate will time out at 1547.154054s 2024-03-29T11:30:10.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dd8f28640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4dd413f970 0x7f4dd4138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:34829/0 (socket says 172.21.15.64:34829) 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dd8f28640 1 -- 172.21.15.64:0/1518671161 learned_addr learned my addr 172.21.15.64:0/1518671161 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 10 client.?.objecter ms_handle_connect 0x7f4dd413f970 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442943824 0 0) 0x7f4dd41390b0 con 0x7f4dd413f970 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.884+0000 7f4dc9ffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.885+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db8004eb0 con 0x7f4dd413f970 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 client.?.objecter ms_handle_connect 0x7f4dd4142800 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2633336535 0 0) 0x7f4db8004eb0 con 0x7f4dd413f970 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dd41390b0 con 0x7f4dd413f970 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 client.?.objecter ms_handle_connect 0x7f4dd40ae250 2024-03-29T11:30:10.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4dcc001db0 con 0x7f4dd413f970 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): dump: 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:10.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346340717 0 0) 0x7f4dd41e47f0 con 0x7f4dd4142800 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.887+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db8004eb0 con 0x7f4dd4142800 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.888+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 798736445 0 0) 0x7f4dd41390b0 con 0x7f4dd413f970 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.888+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 >> v1:172.21.15.64:6790/0 conn(0x7f4dd40ae250 legacy=0x7f4dd4139870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.888+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 >> v1:172.21.15.64:6789/0 conn(0x7f4dd4142800 legacy=0x7f4dd41389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.889+0000 7f4dc9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.889+0000 7f4dc9ffb640 1 monclient: found mon.b 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.889+0000 7f4dc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.889+0000 7f4dc9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd41437f0 con 0x7f4dd413f970 2024-03-29T11:30:10.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dc9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dc9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 10 monclient: _renew_subs 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd4139f80 con 0x7f4dd413f970 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dcc001f10 con 0x7f4dd413f970 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 10 monclient: _renew_subs 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd41e4e60 con 0x7f4dd413f970 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4ddb98d640 1 librados: init done 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.890+0000 7f4dcaffd640 4 set_mon_vals no callback set 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.891+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4dcc003de0 con 0x7f4dd413f970 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.891+0000 7f4dc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.891+0000 7f4dc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.891+0000 7f4dc9ffb640 10 monclient: dump: 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:10.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.892+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4dcc003f40 con 0x7f4dd413f970 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.893+0000 7f4dd9729640 10 monclient: get_auth_request con 0x7f4db806db00 auth_method 0 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.893+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f4dcc084ee0 con 0x7f4dd413f970 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.894+0000 7f4dc9ffb640 10 client.4468.objecter ms_dispatch 0x7f4dd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.894+0000 7f4dc9ffb640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.894+0000 7f4dc9ffb640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.894+0000 7f4dc9ffb640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.894+0000 7f4dc9ffb640 10 client.4468.objecter ms_handle_connect 0x7f4db806db00 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.895+0000 7f4ddb98d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.895+0000 7f4ddb98d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.895+0000 7f4ddb98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.895+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db4005080 con 0x7f4dd413f970 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.902+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4dcc0507c0 con 0x7f4dd413f970 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.902+0000 7f4dc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:10.902+0000 7f4dc9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.231+0000 7f4ddb98d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.231+0000 7f4ddb98d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.231+0000 7f4ddb98d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.231+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db4005330 con 0x7f4dd413f970 2024-03-29T11:30:11.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.233+0000 7f4dc9ffb640 1 -- 172.21.15.64:0/1518671161 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f4dcc0526f0 con 0x7f4dd413f970 2024-03-29T11:30:11.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.233+0000 7f4dc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:11.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.233+0000 7f4dc9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:11.225 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:11.225 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.236+0000 7f4ddb98d640 10 librados: watch_flush enter 2024-03-29T11:30:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.236+0000 7f4ddb98d640 10 librados: watch_flush exit 2024-03-29T11:30:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.236+0000 7f4ddb98d640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.236+0000 7f4ddb98d640 10 client.4468.objecter successfully canceled tick 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 >> v1:172.21.15.136:6816/2312274743 conn(0x7f4db806db00 legacy=0x7f4db806ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 10 monclient: shutdown 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 >> v1:172.21.15.136:6789/0 conn(0x7f4dd413f970 legacy=0x7f4dd4138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 shutdown_connections 2024-03-29T11:30:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 >> 172.21.15.64:0/1518671161 conn(0x7f4dd40b35e0 msgr2=0x7f4dd40aa0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:11.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 shutdown_connections 2024-03-29T11:30:11.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 -- 172.21.15.64:0/1518671161 wait complete. 2024-03-29T11:30:11.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:11.237+0000 7f4ddb98d640 1 librados: shutdown 2024-03-29T11:30:11.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:18.053 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:18.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.210+0000 7f20438e0640 10 monclient: get_monmap_and_config 2024-03-29T11:30:18.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.210+0000 7f20438e0640 10 monclient: build_initial_monmap 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.210+0000 7f20438e0640 10 monclient: monmap: 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:18.211888+0000 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:18.211888+0000 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:18.204 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:18.205 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:18.205 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:18.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.211+0000 7f20438e0640 1 Processor -- start 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.212+0000 7f20438e0640 1 -- start start 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.212+0000 7f20438e0640 10 monclient: init 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient(hunting): picked mon.noname-a con 0x7f203c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient(hunting): picked mon.noname-b con 0x7f203c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient(hunting): picked mon.noname-c con 0x7f203c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:18.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0aaef0 con 0x7f203c1457f0 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c145bd0 con 0x7f203c0a9d70 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c145eb0 con 0x7f203c13f650 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f20438e0640 10 monclient(hunting): authenticate will time out at 1554.483060s 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f2040e7b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f203c13f650 0x7f203c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45428/0 (socket says 172.21.15.64:45428) 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.213+0000 7f2040e7b640 1 -- 172.21.15.64:0/78972833 learned_addr learned my addr 172.21.15.64:0/78972833 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003285601 0 0) 0x7f203c145eb0 con 0x7f203c13f650 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2028003520 con 0x7f203c13f650 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195812179 0 0) 0x7f203c145bd0 con 0x7f203c0a9d70 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c145eb0 con 0x7f203c0a9d70 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11282464 0 0) 0x7f203c0aaef0 con 0x7f203c1457f0 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.214+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c145bd0 con 0x7f203c1457f0 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911590156 0 0) 0x7f203c145bd0 con 0x7f203c1457f0 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 >> v1:172.21.15.64:6790/0 conn(0x7f203c13f650 legacy=0x7f203c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 >> v1:172.21.15.136:6789/0 conn(0x7f203c0a9d70 legacy=0x7f203c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 monclient: found mon.noname-a 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c146340 con 0x7f203c1457f0 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: _finish_auth 0 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2038001570 con 0x7f203c1457f0 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: dump: 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.215 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f20438e0640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f20438e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2038001740 con 0x7f203c1457f0 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 1 -- 172.21.15.64:0/78972833 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2038001b10 con 0x7f203c1457f0 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f2033fff640 10 monclient: dump: 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:18.216 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f20438e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.215+0000 7f20438e0640 4 set_mon_vals no callback set 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.216+0000 7f20438e0640 10 monclient: shutdown 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.216+0000 7f20438e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.216+0000 7f20438e0640 1 -- 172.21.15.64:0/78972833 >> v1:172.21.15.64:6789/0 conn(0x7f203c1457f0 legacy=0x7f203c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.216+0000 7f20438e0640 1 -- 172.21.15.64:0/78972833 shutdown_connections 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.216+0000 7f20438e0640 1 -- 172.21.15.64:0/78972833 >> 172.21.15.64:0/78972833 conn(0x7f203c066a00 msgr2=0x7f203c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.217+0000 7f20438e0640 1 -- 172.21.15.64:0/78972833 shutdown_connections 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.217+0000 7f20438e0640 1 -- 172.21.15.64:0/78972833 wait complete. 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 10 monclient: build_initial_monmap 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 10 monclient: monmap: 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:18.218 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:18.219 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:18.219 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:18.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 1 librados: starting msgr at 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 1 librados: starting objecter 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 1 Processor -- start 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 1 -- start start 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 1 librados: setting wanted keys 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 1 librados: calling monclient init 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.218+0000 7f20438e0640 10 monclient: init 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:18.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:18.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient(hunting): picked mon.noname-a con 0x7f203c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:18.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient(hunting): picked mon.noname-b con 0x7f203c13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:18.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient(hunting): picked mon.noname-c con 0x7f203c1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:18.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c1419e0 con 0x7f203c0a9d70 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0700e0 con 0x7f203c13f650 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c071260 con 0x7f203c1457f0 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.219+0000 7f20438e0640 10 monclient(hunting): authenticate will time out at 1554.489703s 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f204167c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f203c0a9d70 0x7f203c070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41058/0 (socket says 172.21.15.64:41058) 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f204167c640 1 -- 172.21.15.64:0/2411992771 learned_addr learned my addr 172.21.15.64:0/2411992771 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 client.?.objecter ms_handle_connect 0x7f203c0a9d70 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:18.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 client.?.objecter ms_handle_connect 0x7f203c13f650 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 client.?.objecter ms_handle_connect 0x7f203c1457f0 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722452212 0 0) 0x7f203c1419e0 con 0x7f203c0a9d70 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.220+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2020004eb0 con 0x7f203c0a9d70 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840510114 0 0) 0x7f203c071260 con 0x7f203c1457f0 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c1419e0 con 0x7f203c1457f0 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048590613 0 0) 0x7f2020004eb0 con 0x7f203c0a9d70 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f203c071260 con 0x7f203c0a9d70 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2038001d50 con 0x7f203c0a9d70 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): dump: 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:18.227 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951665928 0 0) 0x7f203c0700e0 con 0x7f203c13f650 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:30:18.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2020004eb0 con 0x7f203c13f650 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3255362700 0 0) 0x7f203c071260 con 0x7f203c0a9d70 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 >> v1:172.21.15.64:6790/0 conn(0x7f203c1457f0 legacy=0x7f203c1e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 >> v1:172.21.15.136:6789/0 conn(0x7f203c13f650 legacy=0x7f203c1402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 monclient: found mon.a 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c142c00 con 0x7f203c0a9d70 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:18.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f2031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f20438e0640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f20438e0640 10 monclient: _renew_subs 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f20438e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.221+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c071ec0 con 0x7f203c0a9d70 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f20438e0640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f20438e0640 10 monclient: _renew_subs 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f20438e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:18.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c13c8d0 con 0x7f203c0a9d70 2024-03-29T11:30:18.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f20438e0640 1 librados: init done 2024-03-29T11:30:18.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20380038a0 con 0x7f203c0a9d70 2024-03-29T11:30:18.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f2031ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:18.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2038003c30 con 0x7f203c0a9d70 2024-03-29T11:30:18.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f2031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f2031ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f2031ffb640 10 monclient: dump: 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.222+0000 7f20337fe640 4 set_mon_vals no callback set 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.223+0000 7f20438e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.224+0000 7f20438e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:18.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.224+0000 7f20438e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:18.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.224+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f2038003d90 con 0x7f203c0a9d70 2024-03-29T11:30:18.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.224+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201c005080 con 0x7f203c0a9d70 2024-03-29T11:30:18.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f2038084b00 con 0x7f203c0a9d70 2024-03-29T11:30:18.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2031ffb640 10 client.4461.objecter ms_dispatch 0x7f203c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:18.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2031ffb640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:18.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2031ffb640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:18.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2031ffb640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:30:18.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2040e7b640 10 monclient: get_auth_request con 0x7f202006de50 auth_method 0 2024-03-29T11:30:18.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.225+0000 7f2031ffb640 10 client.4461.objecter ms_handle_connect 0x7f202006de50 2024-03-29T11:30:18.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.234+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20380503e0 con 0x7f203c0a9d70 2024-03-29T11:30:18.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.234+0000 7f2031ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:18.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.234+0000 7f2031ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:18.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.614+0000 7f20438e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.614+0000 7f20438e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.614+0000 7f20438e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:18.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.614+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f201c005330 con 0x7f203c0a9d70 2024-03-29T11:30:18.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.615+0000 7f2031ffb640 1 -- 172.21.15.64:0/2411992771 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f2038052310 con 0x7f203c0a9d70 2024-03-29T11:30:18.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.615+0000 7f2031ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.615+0000 7f2031ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:18.608 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:18.608 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:18.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.618+0000 7f20438e0640 10 librados: watch_flush enter 2024-03-29T11:30:18.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.618+0000 7f20438e0640 10 librados: watch_flush exit 2024-03-29T11:30:18.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.618+0000 7f20438e0640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:30:18.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.618+0000 7f20438e0640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:30:18.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 >> v1:172.21.15.136:6816/2312274743 conn(0x7f202006de50 legacy=0x7f2020070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 10 monclient: shutdown 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 >> v1:172.21.15.64:6789/0 conn(0x7f203c0a9d70 legacy=0x7f203c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 shutdown_connections 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 >> 172.21.15.64:0/2411992771 conn(0x7f203c066a00 msgr2=0x7f203c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 shutdown_connections 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 -- 172.21.15.64:0/2411992771 wait complete. 2024-03-29T11:30:18.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:18.619+0000 7f20438e0640 1 librados: shutdown 2024-03-29T11:30:18.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:25.434 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:25.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.589+0000 7f8534863640 10 monclient: get_monmap_and_config 2024-03-29T11:30:25.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.589+0000 7f8534863640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.589+0000 7f8534863640 10 monclient: monmap: 2024-03-29T11:30:25.583 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:25.583 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:25.591523+0000 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:25.591523+0000 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:25.584 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:25.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.590+0000 7f8534863640 1 Processor -- start 2024-03-29T11:30:25.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.591+0000 7f8534863640 1 -- start start 2024-03-29T11:30:25.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.591+0000 7f8534863640 10 monclient: init 2024-03-29T11:30:25.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:25.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient(hunting): picked mon.noname-c con 0x7f85300b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient(hunting): picked mon.noname-a con 0x7f8530071eb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient(hunting): picked mon.noname-b con 0x7f8530143d10 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853006ef80 con 0x7f8530071eb0 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853006f260 con 0x7f85300b1f60 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85300b40c0 con 0x7f8530143d10 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f8534863640 10 monclient(hunting): authenticate will time out at 1561.862985s 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f852ed9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f8530143d10 0x7f8530146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39998/0 (socket says 172.21.15.64:39998) 2024-03-29T11:30:25.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.592+0000 7f852ed9d640 1 -- 172.21.15.64:0/1331434388 learned_addr learned my addr 172.21.15.64:0/1331434388 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:25.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.593+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080491136 0 0) 0x7f85300b40c0 con 0x7f8530143d10 2024-03-29T11:30:25.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.593+0000 7f852d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.593+0000 7f852d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.594+0000 7f852d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.594+0000 7f852d59a640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:30:25.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.594+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8518003520 con 0x7f8530143d10 2024-03-29T11:30:25.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.595+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3010730015 0 0) 0x7f8518003520 con 0x7f8530143d10 2024-03-29T11:30:25.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.595+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 >> v1:172.21.15.136:6789/0 conn(0x7f85300b1f60 legacy=0x7f8530143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.597+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 >> v1:172.21.15.64:6789/0 conn(0x7f8530071eb0 legacy=0x7f8530146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.597+0000 7f852d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.597+0000 7f852d59a640 1 monclient: found mon.noname-b 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.597+0000 7f852d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.597+0000 7f852d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.597+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530155d70 con 0x7f8530143d10 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f8534863640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f8534863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:25.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8528001a10 con 0x7f8530143d10 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 10 monclient: dump: 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:25.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f8534863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8528001ca0 con 0x7f8530143d10 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f852d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f8534863640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.598+0000 7f8534863640 4 set_mon_vals no callback set 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f852d59a640 1 -- 172.21.15.64:0/1331434388 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8528002070 con 0x7f8530143d10 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f852d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f852d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f852d59a640 10 monclient: dump: 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:25.595 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:25.596 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:25.596 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:25.596 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f8534863640 10 monclient: shutdown 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f8534863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f8534863640 1 -- 172.21.15.64:0/1331434388 >> v1:172.21.15.64:6790/0 conn(0x7f8530143d10 legacy=0x7f8530146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f8534863640 1 -- 172.21.15.64:0/1331434388 shutdown_connections 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.599+0000 7f8534863640 1 -- 172.21.15.64:0/1331434388 >> 172.21.15.64:0/1331434388 conn(0x7f8530066a00 msgr2=0x7f8530066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.600+0000 7f8534863640 1 -- 172.21.15.64:0/1331434388 shutdown_connections 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.600+0000 7f8534863640 1 -- 172.21.15.64:0/1331434388 wait complete. 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.600+0000 7f8534863640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.600+0000 7f8534863640 10 monclient: monmap: 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 1 librados: starting msgr at 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 1 librados: starting objecter 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 1 Processor -- start 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 1 -- start start 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 1 librados: setting wanted keys 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 1 librados: calling monclient init 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.601+0000 7f8534863640 10 monclient: init 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient(hunting): picked mon.noname-b con 0x7f85300b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient(hunting): picked mon.noname-a con 0x7f8530143d10 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient(hunting): picked mon.noname-c con 0x7f8530141090 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8530137630 con 0x7f8530143d10 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85301423d0 con 0x7f85300b1f60 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85301540c0 con 0x7f8530141090 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.602+0000 7f8534863640 10 monclient(hunting): authenticate will time out at 1561.873420s 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f852ed9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f8530141090 0x7f85301e6ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42928/0 (socket says 172.21.15.64:42928) 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f852ed9d640 1 -- 172.21.15.64:0/299798577 learned_addr learned my addr 172.21.15.64:0/299798577 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 10 client.?.objecter ms_handle_connect 0x7f8530141090 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989520217 0 0) 0x7f85301540c0 con 0x7f8530141090 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.603+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8510004eb0 con 0x7f8530141090 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.604+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029833688 0 0) 0x7f8510004eb0 con 0x7f8530141090 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.604+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85301540c0 con 0x7f8530141090 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.604+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8528001990 con 0x7f8530141090 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.604+0000 7f851effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.604+0000 7f851effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.604+0000 7f851effd640 10 monclient(hunting): dump: 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.604 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.605+0000 7f851effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.605+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 794164384 0 0) 0x7f85301540c0 con 0x7f8530141090 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.605+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 >> v1:172.21.15.136:6789/0 conn(0x7f85300b1f60 legacy=0x7f85301539b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 >> v1:172.21.15.64:6789/0 conn(0x7f8530143d10 legacy=0x7f85301547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 1 monclient: found mon.c 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530136ad0 con 0x7f8530141090 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f851effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 10 monclient: _renew_subs 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8530136d20 con 0x7f8530141090 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 10 monclient: _renew_subs 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8530137ca0 con 0x7f8530141090 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.606+0000 7f8534863640 1 librados: init done 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f8534863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f8534863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f8534863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8530142210 con 0x7f8530141090 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8528001af0 con 0x7f8530141090 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f851effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.608+0000 7f852cd99640 4 set_mon_vals no callback set 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.609+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f8528005070 con 0x7f8530141090 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.609+0000 7f851effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.609+0000 7f851effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.609+0000 7f851effd640 10 monclient: dump: 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:25.610 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.610+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f85280051d0 con 0x7f8530141090 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.611+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f8528084ef0 con 0x7f8530141090 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.611+0000 7f851effd640 10 client.4478.objecter ms_dispatch 0x7f8530000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.611+0000 7f851effd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.611+0000 7f851effd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.611+0000 7f852e59c640 10 monclient: get_auth_request con 0x7f851006dce0 auth_method 0 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.612+0000 7f851effd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:30:25.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.612+0000 7f851effd640 10 client.4478.objecter ms_handle_connect 0x7f851006dce0 2024-03-29T11:30:25.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.617+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8528050600 con 0x7f8530141090 2024-03-29T11:30:25.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.617+0000 7f851effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.617+0000 7f851effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.953+0000 7f8534863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.953+0000 7f8534863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.953+0000 7f8534863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:25.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.953+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85301377e0 con 0x7f8530141090 2024-03-29T11:30:25.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.954+0000 7f851effd640 1 -- 172.21.15.64:0/299798577 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f85301377e0 con 0x7f8530141090 2024-03-29T11:30:25.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.954+0000 7f851effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.954+0000 7f851effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:25.948 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:25.948 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.957+0000 7f8534863640 10 librados: watch_flush enter 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.958+0000 7f8534863640 10 librados: watch_flush exit 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.958+0000 7f8534863640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.958+0000 7f8534863640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.958+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 >> v1:172.21.15.136:6816/2312274743 conn(0x7f851006dce0 legacy=0x7f85100701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 10 monclient: shutdown 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 >> v1:172.21.15.64:6790/0 conn(0x7f8530141090 legacy=0x7f85301e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 shutdown_connections 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 >> 172.21.15.64:0/299798577 conn(0x7f8530066a00 msgr2=0x7f85300b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 shutdown_connections 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 1 -- 172.21.15.64:0/299798577 wait complete. 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:25.959+0000 7f8534863640 1 librados: shutdown 2024-03-29T11:30:25.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:32.775 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.941+0000 7f3530bcf640 10 monclient: get_monmap_and_config 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.941+0000 7f3530bcf640 10 monclient: build_initial_monmap 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.941+0000 7f3530bcf640 10 monclient: monmap: 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:32.942274+0000 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:32.942274+0000 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:32.934 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.942+0000 7f3530bcf640 1 Processor -- start 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.942+0000 7f3530bcf640 1 -- start start 2024-03-29T11:30:32.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.942+0000 7f3530bcf640 10 monclient: init 2024-03-29T11:30:32.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:32.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:32.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f352c0b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:32.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f352c071eb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:32.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f352c143d10 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:32.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c06ef80 con 0x7f352c071eb0 2024-03-29T11:30:32.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c06f260 con 0x7f352c0b1f60 2024-03-29T11:30:32.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0b40c0 con 0x7f352c143d10 2024-03-29T11:30:32.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:32.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.943+0000 7f3530bcf640 10 monclient(hunting): authenticate will time out at 1569.213704s 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352a59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f352c0b1f60 0x7f352c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:34774/0 (socket says 172.21.15.64:34774) 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352a59c640 1 -- 172.21.15.64:0/676545149 learned_addr learned my addr 172.21.15.64:0/676545149 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277309298 0 0) 0x7f352c06f260 con 0x7f352c0b1f60 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352959a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352959a640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.944+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514003520 con 0x7f352c0b1f60 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265319142 0 0) 0x7f352c0b40c0 con 0x7f352c143d10 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:30:32.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c06f260 con 0x7f352c143d10 2024-03-29T11:30:32.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 519965341 0 0) 0x7f3514003520 con 0x7f352c0b1f60 2024-03-29T11:30:32.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 >> v1:172.21.15.64:6790/0 conn(0x7f352c143d10 legacy=0x7f352c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.945+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 >> v1:172.21.15.64:6789/0 conn(0x7f352c071eb0 legacy=0x7f352c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:32.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:32.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 1 monclient: found mon.noname-c 2024-03-29T11:30:32.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:32.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:30:32.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c155d70 con 0x7f352c0b1f60 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: _finish_auth 0 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f3530bcf640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f3530bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3520001690 con 0x7f352c0b1f60 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: dump: 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.944 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f3530bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35200017f0 con 0x7f352c0b1f60 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f3530bcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f3530bcf640 4 set_mon_vals no callback set 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 1 -- 172.21.15.64:0/676545149 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3520001c40 con 0x7f352c0b1f60 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.948+0000 7f352959a640 10 monclient: dump: 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:32.945 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 10 monclient: shutdown 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 1 -- 172.21.15.64:0/676545149 >> v1:172.21.15.136:6789/0 conn(0x7f352c0b1f60 legacy=0x7f352c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 1 -- 172.21.15.64:0/676545149 shutdown_connections 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 1 -- 172.21.15.64:0/676545149 >> 172.21.15.64:0/676545149 conn(0x7f352c066a00 msgr2=0x7f352c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 1 -- 172.21.15.64:0/676545149 shutdown_connections 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.949+0000 7f3530bcf640 1 -- 172.21.15.64:0/676545149 wait complete. 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.950+0000 7f3530bcf640 10 monclient: build_initial_monmap 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.950+0000 7f3530bcf640 10 monclient: monmap: 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 1 librados: starting msgr at 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 1 librados: starting objecter 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 1 Processor -- start 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 1 -- start start 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 1 librados: setting wanted keys 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 1 librados: calling monclient init 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.951+0000 7f3530bcf640 10 monclient: init 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.952+0000 7f3530bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.952+0000 7f3530bcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.952+0000 7f3530bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f352c0b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.952+0000 7f3530bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f352c143d10 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.952+0000 7f352a59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f352c0b1f60 0x7f352c0ab0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54448/0 (socket says 172.21.15.64:54448) 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.952+0000 7f3529d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f352c143d10 0x7f352c1e42a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:36904/0 (socket says 172.21.15.64:36904) 2024-03-29T11:30:32.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f3530bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f352c1e49b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f3530bcf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0ac6a0 con 0x7f352c1e49b0 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f3530bcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0ac980 con 0x7f352c143d10 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f3530bcf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0aa320 con 0x7f352c0b1f60 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f3530bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f3530bcf640 10 monclient(hunting): authenticate will time out at 1569.223383s 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f352a59c640 1 -- 172.21.15.64:0/3141931331 learned_addr learned my addr 172.21.15.64:0/3141931331 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f351affd640 10 client.?.objecter ms_handle_connect 0x7f352c0b1f60 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f351affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f351affd640 10 client.?.objecter ms_handle_connect 0x7f352c143d10 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.953+0000 7f351affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670382818 0 0) 0x7f352c0aa320 con 0x7f352c0b1f60 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350c004eb0 con 0x7f352c0b1f60 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429416137 0 0) 0x7f352c0ac980 con 0x7f352c143d10 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.954+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c0aa320 con 0x7f352c143d10 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3595226532 0 0) 0x7f350c004eb0 con 0x7f352c0b1f60 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f352c0ac980 con 0x7f352c0b1f60 2024-03-29T11:30:32.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3520001b00 con 0x7f352c0b1f60 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): dump: 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:32.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000791425 0 0) 0x7f352c0aa320 con 0x7f352c143d10 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350c004eb0 con 0x7f352c143d10 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f351c001c70 con 0x7f352c143d10 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient(hunting): dump: 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:32.959 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 847466663 0 0) 0x7f350c004eb0 con 0x7f352c143d10 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 >> v1:172.21.15.64:6790/0 conn(0x7f352c0b1f60 legacy=0x7f352c0ab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 >> v1:172.21.15.64:6789/0 conn(0x7f352c1e49b0 legacy=0x7f352c13fc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 monclient: found mon.b 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c1e3d80 con 0x7f352c143d10 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient: _finish_auth 0 2024-03-29T11:30:32.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f3530bcf640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f3530bcf640 10 monclient: _renew_subs 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f3530bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c0b1620 con 0x7f352c143d10 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.955+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f351c003990 con 0x7f352c143d10 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f351affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f351c003ce0 con 0x7f352c143d10 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f351affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f351affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f351affd640 10 monclient: dump: 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:32.961 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f3528d99640 4 set_mon_vals no callback set 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f3530bcf640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f3530bcf640 10 monclient: _renew_subs 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f3530bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:32.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c140360 con 0x7f352c143d10 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.956+0000 7f3530bcf640 1 librados: init done 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f351c003e40 con 0x7f352c143d10 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f3530bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f3530bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f3530bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352c0ab7b0 con 0x7f352c143d10 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f351c084ce0 con 0x7f352c143d10 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f351affd640 10 client.4480.objecter ms_dispatch 0x7f352c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f351affd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.958+0000 7f351affd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.959+0000 7f351affd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.959+0000 7f352a59c640 10 monclient: get_auth_request con 0x7f350c06dec0 auth_method 0 2024-03-29T11:30:32.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.959+0000 7f351affd640 10 client.4480.objecter ms_handle_connect 0x7f350c06dec0 2024-03-29T11:30:32.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.966+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f351c0505c0 con 0x7f352c143d10 2024-03-29T11:30:32.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.966+0000 7f351affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:32.966+0000 7f351affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:33.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.299+0000 7f3530bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.299+0000 7f3530bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.299+0000 7f3530bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.299+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f352c075f10 con 0x7f352c143d10 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.301+0000 7f351affd640 1 -- 172.21.15.64:0/3141931331 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f351c0524f0 con 0x7f352c143d10 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.301+0000 7f351affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.301+0000 7f351affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:33.294 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:33.294 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.304+0000 7f3530bcf640 10 librados: watch_flush enter 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.304+0000 7f3530bcf640 10 librados: watch_flush exit 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.304+0000 7f3530bcf640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.304+0000 7f3530bcf640 10 client.4480.objecter successfully canceled tick 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.305+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 >> v1:172.21.15.136:6816/2312274743 conn(0x7f350c06dec0 legacy=0x7f350c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.305+0000 7f3530bcf640 10 monclient: shutdown 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.305+0000 7f3530bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.305+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 >> v1:172.21.15.136:6789/0 conn(0x7f352c143d10 legacy=0x7f352c1e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.305+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 shutdown_connections 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.305+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 >> 172.21.15.64:0/3141931331 conn(0x7f352c066a00 msgr2=0x7f352c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.306+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 shutdown_connections 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.306+0000 7f3530bcf640 1 -- 172.21.15.64:0/3141931331 wait complete. 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:33.306+0000 7f3530bcf640 1 librados: shutdown 2024-03-29T11:30:33.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:40.122 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:40.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.285+0000 7f3b8bcbb640 10 monclient: get_monmap_and_config 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.285+0000 7f3b8bcbb640 10 monclient: build_initial_monmap 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.285+0000 7f3b8bcbb640 10 monclient: monmap: 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:40.286538+0000 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:40.286538+0000 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:40.279 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.286+0000 7f3b8bcbb640 1 Processor -- start 2024-03-29T11:30:40.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.287+0000 7f3b8bcbb640 1 -- start start 2024-03-29T11:30:40.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.287+0000 7f3b8bcbb640 10 monclient: init 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient(hunting): picked mon.noname-b con 0x7f3b840b0770 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient(hunting): picked mon.noname-a con 0x7f3b8414a700 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient(hunting): picked mon.noname-c con 0x7f3b840ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b840aba70 con 0x7f3b8414a700 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b840aecb0 con 0x7f3b840ac5a0 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b840af150 con 0x7f3b840b0770 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:40.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b8bcbb640 10 monclient(hunting): authenticate will time out at 1576.558233s 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b89a57640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3b840b0770 0x7f3b84149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59342/0 (socket says 172.21.15.64:59342) 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.288+0000 7f3b89a57640 1 -- 172.21.15.64:0/819755101 learned_addr learned my addr 172.21.15.64:0/819755101 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461681407 0 0) 0x7f3b840af150 con 0x7f3b840b0770 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b74003520 con 0x7f3b840b0770 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734786965 0 0) 0x7f3b840aba70 con 0x7f3b8414a700 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:30:40.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b840af150 con 0x7f3b8414a700 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148569261 0 0) 0x7f3b840af150 con 0x7f3b8414a700 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 >> v1:172.21.15.64:6790/0 conn(0x7f3b840b0770 legacy=0x7f3b84149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 >> v1:172.21.15.136:6789/0 conn(0x7f3b840ac5a0 legacy=0x7f3b84151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: _finish_hunting 0 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 monclient: found mon.noname-a 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840ac9d0 con 0x7f3b8414a700 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: _finish_auth 0 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: _check_auth_tickets 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b70001630 con 0x7f3b8414a700 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b88a55640 10 monclient: dump: 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b8bcbb640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.289+0000 7f3b8bcbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:40.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b70001880 con 0x7f3b8414a700 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b88a55640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b8bcbb640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b8bcbb640 4 set_mon_vals no callback set 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b88a55640 1 -- 172.21.15.64:0/819755101 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b70001630 con 0x7f3b8414a700 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b88a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b88a55640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.290+0000 7f3b88a55640 10 monclient: dump: 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.286 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:40.287 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:40.287 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:40.287 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 10 monclient: shutdown 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/819755101 >> v1:172.21.15.64:6789/0 conn(0x7f3b8414a700 legacy=0x7f3b840abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/819755101 shutdown_connections 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/819755101 >> 172.21.15.64:0/819755101 conn(0x7f3b8406b660 msgr2=0x7f3b8406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/819755101 shutdown_connections 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.291+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/819755101 wait complete. 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.292+0000 7f3b8bcbb640 10 monclient: build_initial_monmap 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.292+0000 7f3b8bcbb640 10 monclient: monmap: 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:40.291 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.292+0000 7f3b8bcbb640 1 librados: starting msgr at 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.292+0000 7f3b8bcbb640 1 librados: starting objecter 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.293+0000 7f3b8bcbb640 1 Processor -- start 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.293+0000 7f3b8bcbb640 1 -- start start 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.293+0000 7f3b8bcbb640 1 librados: setting wanted keys 2024-03-29T11:30:40.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.293+0000 7f3b8bcbb640 1 librados: calling monclient init 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.293+0000 7f3b8bcbb640 10 monclient: init 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient(hunting): picked mon.noname-a con 0x7f3b840ac5a0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient(hunting): picked mon.noname-c con 0x7f3b840b0770 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient(hunting): picked mon.noname-b con 0x7f3b8414a700 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8412ae50 con 0x7f3b840ac5a0 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b89a57640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3b840ac5a0 0x7f3b841dbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40608/0 (socket says 172.21.15.64:40608) 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b89a57640 1 -- 172.21.15.64:0/963724712 learned_addr learned my addr 172.21.15.64:0/963724712 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b84128ad0 con 0x7f3b8414a700 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b841db740 con 0x7f3b840b0770 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b8bcbb640 10 monclient(hunting): authenticate will time out at 1576.564529s 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 10 client.?.objecter ms_handle_connect 0x7f3b840ac5a0 2024-03-29T11:30:40.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074401851 0 0) 0x7f3b8412ae50 con 0x7f3b840ac5a0 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.294+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b68004eb0 con 0x7f3b840ac5a0 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 client.?.objecter ms_handle_connect 0x7f3b8414a700 2024-03-29T11:30:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3630463693 0 0) 0x7f3b68004eb0 con 0x7f3b840ac5a0 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b8412ae50 con 0x7f3b840ac5a0 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 client.?.objecter ms_handle_connect 0x7f3b840b0770 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b78001cd0 con 0x7f3b840ac5a0 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): dump: 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:40.295 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007308887 0 0) 0x7f3b84128ad0 con 0x7f3b8414a700 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b68004eb0 con 0x7f3b8414a700 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105082360 0 0) 0x7f3b841db740 con 0x7f3b840b0770 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.295+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b84128ad0 con 0x7f3b840b0770 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682730136 0 0) 0x7f3b68004eb0 con 0x7f3b8414a700 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b841db740 con 0x7f3b8414a700 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b7c001490 con 0x7f3b8414a700 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient(hunting): dump: 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 53616613 0 0) 0x7f3b8412ae50 con 0x7f3b840ac5a0 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 >> v1:172.21.15.64:6790/0 conn(0x7f3b840b0770 legacy=0x7f3b84147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 >> v1:172.21.15.136:6789/0 conn(0x7f3b8414a700 legacy=0x7f3b841e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 monclient: found mon.a 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b84126140 con 0x7f3b840ac5a0 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 10 monclient: _renew_subs 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b841dc370 con 0x7f3b840ac5a0 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 10 monclient: _renew_subs 2024-03-29T11:30:40.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b841291d0 con 0x7f3b840ac5a0 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.296+0000 7f3b8bcbb640 1 librados: init done 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b78004b60 con 0x7f3b840ac5a0 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b78004f70 con 0x7f3b840ac5a0 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b83fff640 4 set_mon_vals no callback set 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b827fc640 10 monclient: dump: 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.297+0000 7f3b8bcbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.298+0000 7f3b8bcbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.298+0000 7f3b8bcbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:40.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.298+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c005080 con 0x7f3b840ac5a0 2024-03-29T11:30:40.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.299+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f3b7801af40 con 0x7f3b840ac5a0 2024-03-29T11:30:40.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.300+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f3b78084e60 con 0x7f3b840ac5a0 2024-03-29T11:30:40.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.300+0000 7f3b827fc640 10 client.4473.objecter ms_dispatch 0x7f3b84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:40.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.300+0000 7f3b827fc640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:40.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.300+0000 7f3b827fc640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:40.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.300+0000 7f3b89256640 10 monclient: get_auth_request con 0x7f3b6806ddd0 auth_method 0 2024-03-29T11:30:40.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.300+0000 7f3b827fc640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:30:40.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.301+0000 7f3b827fc640 10 client.4473.objecter ms_handle_connect 0x7f3b6806ddd0 2024-03-29T11:30:40.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.306+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b78050780 con 0x7f3b840ac5a0 2024-03-29T11:30:40.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.306+0000 7f3b827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.306+0000 7f3b827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.670+0000 7f3b8bcbb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.670+0000 7f3b8bcbb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.670+0000 7f3b8bcbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:30:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.670+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b6c005330 con 0x7f3b840ac5a0 2024-03-29T11:30:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.671+0000 7f3b827fc640 1 -- 172.21.15.64:0/963724712 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f3b780526b0 con 0x7f3b840ac5a0 2024-03-29T11:30:40.664 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:40.664 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.671+0000 7f3b827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.671+0000 7f3b827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:40.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 10 librados: watch_flush enter 2024-03-29T11:30:40.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 10 librados: watch_flush exit 2024-03-29T11:30:40.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:30:40.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 >> v1:172.21.15.136:6816/2312274743 conn(0x7f3b6806ddd0 legacy=0x7f3b68070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 10 monclient: shutdown 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 >> v1:172.21.15.64:6789/0 conn(0x7f3b840ac5a0 legacy=0x7f3b841dbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 shutdown_connections 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.674+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 >> 172.21.15.64:0/963724712 conn(0x7f3b8406b660 msgr2=0x7f3b8406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.675+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 shutdown_connections 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.675+0000 7f3b8bcbb640 1 -- 172.21.15.64:0/963724712 wait complete. 2024-03-29T11:30:40.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:40.675+0000 7f3b8bcbb640 1 librados: shutdown 2024-03-29T11:30:40.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:47.491 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:47.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.633+0000 7fdf177c8640 10 monclient: get_monmap_and_config 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.633+0000 7fdf177c8640 10 monclient: build_initial_monmap 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.633+0000 7fdf177c8640 10 monclient: monmap: 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:47.634638+0000 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:47.634638+0000 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:47.627 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:47.628 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:47.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.634+0000 7fdf177c8640 1 Processor -- start 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.634+0000 7fdf177c8640 1 -- start start 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.634+0000 7fdf177c8640 10 monclient: init 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdf101457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdf1013f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdf100a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf100aaef0 con 0x7fdf101457f0 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf10145bd0 con 0x7fdf1013f650 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf10145eb0 con 0x7fdf100a9d70 2024-03-29T11:30:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf177c8640 10 monclient(hunting): authenticate will time out at 1583.905421s 2024-03-29T11:30:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf14d63640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fdf1013f650 0x7fdf100b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:50228/0 (socket says 172.21.15.64:50228) 2024-03-29T11:30:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.635+0000 7fdf14d63640 1 -- 172.21.15.64:0/673997756 learned_addr learned my addr 172.21.15.64:0/673997756 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:47.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570165204 0 0) 0x7fdf10145eb0 con 0x7fdf100a9d70 2024-03-29T11:30:47.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc003520 con 0x7fdf100a9d70 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537604762 0 0) 0x7fdf10145bd0 con 0x7fdf1013f650 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf10145eb0 con 0x7fdf1013f650 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031579644 0 0) 0x7fdefc003520 con 0x7fdf100a9d70 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 >> v1:172.21.15.136:6789/0 conn(0x7fdf1013f650 legacy=0x7fdf100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.636+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 >> v1:172.21.15.64:6789/0 conn(0x7fdf101457f0 legacy=0x7fdf1013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:47.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 1 monclient: found mon.noname-b 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10146340 con 0x7fdf100a9d70 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: _finish_auth 0 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdf08001a10 con 0x7fdf100a9d70 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: dump: 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:47.637 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf177c8640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf177c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf08001c60 con 0x7fdf100a9d70 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 1 -- 172.21.15.64:0/673997756 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdf08001ff0 con 0x7fdf100a9d70 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf07fff640 10 monclient: dump: 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf177c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.637+0000 7fdf177c8640 4 set_mon_vals no callback set 2024-03-29T11:30:47.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.638+0000 7fdf177c8640 10 monclient: shutdown 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.638+0000 7fdf177c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.638+0000 7fdf177c8640 1 -- 172.21.15.64:0/673997756 >> v1:172.21.15.64:6790/0 conn(0x7fdf100a9d70 legacy=0x7fdf1014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.639+0000 7fdf177c8640 1 -- 172.21.15.64:0/673997756 shutdown_connections 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.639+0000 7fdf177c8640 1 -- 172.21.15.64:0/673997756 >> 172.21.15.64:0/673997756 conn(0x7fdf10066a00 msgr2=0x7fdf10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.639+0000 7fdf177c8640 1 -- 172.21.15.64:0/673997756 shutdown_connections 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.639+0000 7fdf177c8640 1 -- 172.21.15.64:0/673997756 wait complete. 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.640+0000 7fdf177c8640 10 monclient: build_initial_monmap 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.640+0000 7fdf177c8640 10 monclient: monmap: 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.640+0000 7fdf177c8640 1 librados: starting msgr at 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.640+0000 7fdf177c8640 1 librados: starting objecter 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 1 Processor -- start 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 1 -- start start 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 1 librados: setting wanted keys 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 1 librados: calling monclient init 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 10 monclient: init 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.641+0000 7fdf177c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdf100a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdf1013f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:47.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdf101457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:47.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf100af3e0 con 0x7fdf101457f0 2024-03-29T11:30:47.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf100adcb0 con 0x7fdf100a9d70 2024-03-29T11:30:47.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf100ae1d0 con 0x7fdf1013f650 2024-03-29T11:30:47.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:47.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.642+0000 7fdf177c8640 10 monclient(hunting): authenticate will time out at 1583.912442s 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf14d63640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdf1013f650 0x7fdf101399a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40948/0 (socket says 172.21.15.64:40948) 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf14d63640 1 -- 172.21.15.64:0/1202906991 learned_addr learned my addr 172.21.15.64:0/1202906991 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 10 client.?.objecter ms_handle_connect 0x7fdf1013f650 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535875734 0 0) 0x7fdf100ae1d0 con 0x7fdf1013f650 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.643+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef4004eb0 con 0x7fdf1013f650 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281800976 0 0) 0x7fdef4004eb0 con 0x7fdf1013f650 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf100ae1d0 con 0x7fdf1013f650 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdf000019f0 con 0x7fdf1013f650 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): dump: 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 client.?.objecter ms_handle_connect 0x7fdf100a9d70 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239140652 0 0) 0x7fdf100adcb0 con 0x7fdf100a9d70 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef4004eb0 con 0x7fdf100a9d70 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2126968002 0 0) 0x7fdf100ae1d0 con 0x7fdf1013f650 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 >> v1:172.21.15.136:6789/0 conn(0x7fdf100a9d70 legacy=0x7fdf101da6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 >> v1:172.21.15.64:6789/0 conn(0x7fdf101457f0 legacy=0x7fdf1013a0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 monclient: found mon.c 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf100ae360 con 0x7fdf1013f650 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.644+0000 7fdf05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf00001b50 con 0x7fdf1013f650 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fdf00004c30 con 0x7fdf1013f650 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf05ffb640 10 monclient: dump: 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf177c8640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf177c8640 10 monclient: _renew_subs 2024-03-29T11:30:47.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf177c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf100ad790 con 0x7fdf1013f650 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.645+0000 7fdf077fe640 4 set_mon_vals no callback set 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.646+0000 7fdf177c8640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.646+0000 7fdf177c8640 10 monclient: _renew_subs 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.646+0000 7fdf177c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.646+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf101dafb0 con 0x7fdf1013f650 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.646+0000 7fdf177c8640 1 librados: init done 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.647+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fdf0001ac00 con 0x7fdf1013f650 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fdf00085cc0 con 0x7fdf1013f650 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf05ffb640 10 client.4496.objecter ms_dispatch 0x7fdf10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf05ffb640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf05ffb640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf05ffb640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:30:47.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf177c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf177c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf177c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:47.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf10075f10 con 0x7fdf1013f650 2024-03-29T11:30:47.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.648+0000 7fdf15564640 10 monclient: get_auth_request con 0x7fdef406dc90 auth_method 0 2024-03-29T11:30:47.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.649+0000 7fdf05ffb640 10 client.4496.objecter ms_handle_connect 0x7fdef406dc90 2024-03-29T11:30:47.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.656+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf00050570 con 0x7fdf1013f650 2024-03-29T11:30:47.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.656+0000 7fdf05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.656+0000 7fdf05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.975+0000 7fdf177c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.975+0000 7fdf177c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.975+0000 7fdf177c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.975+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf100ada90 con 0x7fdf1013f650 2024-03-29T11:30:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.976+0000 7fdf05ffb640 1 -- 172.21.15.64:0/1202906991 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fdf000524a0 con 0x7fdf1013f650 2024-03-29T11:30:47.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.976+0000 7fdf05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.976+0000 7fdf05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:47.970 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:47.970 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:47.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 10 librados: watch_flush enter 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 10 librados: watch_flush exit 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 >> v1:172.21.15.136:6816/2312274743 conn(0x7fdef406dc90 legacy=0x7fdef4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 10 monclient: shutdown 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:47.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.979+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 >> v1:172.21.15.64:6790/0 conn(0x7fdf1013f650 legacy=0x7fdf101399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.980+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 shutdown_connections 2024-03-29T11:30:47.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.980+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 >> 172.21.15.64:0/1202906991 conn(0x7fdf10066a00 msgr2=0x7fdf10066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:47.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.980+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 shutdown_connections 2024-03-29T11:30:47.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.980+0000 7fdf177c8640 1 -- 172.21.15.64:0/1202906991 wait complete. 2024-03-29T11:30:47.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:47.980+0000 7fdf177c8640 1 librados: shutdown 2024-03-29T11:30:47.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:54.794 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.954+0000 7f4b8848f640 10 monclient: get_monmap_and_config 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.954+0000 7f4b8848f640 10 monclient: build_initial_monmap 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.954+0000 7f4b8848f640 10 monclient: monmap: 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:30:54.955857+0000 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:30:54.955857+0000 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:54.948 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:54.949 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:30:54.949 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:30:54.949 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:54.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.957+0000 7f4b8848f640 1 Processor -- start 2024-03-29T11:30:54.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.957+0000 7f4b8848f640 1 -- start start 2024-03-29T11:30:54.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.957+0000 7f4b8848f640 10 monclient: init 2024-03-29T11:30:54.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.958+0000 7f4b8848f640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:54.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.958+0000 7f4b8848f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:54.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.958+0000 7f4b8848f640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8006bfa0 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:54.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8006c380 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:54.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 10 monclient(hunting): picked mon.noname-c con 0x7f4b8014a050 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:54.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8006c870 con 0x7f4b8006c380 2024-03-29T11:30:54.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8006dc10 con 0x7f4b8014a050 2024-03-29T11:30:54.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8006cfd0 con 0x7f4b8006bfa0 2024-03-29T11:30:54.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:54.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.959+0000 7f4b8848f640 10 monclient(hunting): authenticate will time out at 1591.228944s 2024-03-29T11:30:54.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.962+0000 7f4b85a2a640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4b8006c380 0x7f4b80071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55547/0 (socket says 172.21.15.64:55547) 2024-03-29T11:30:54.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.962+0000 7f4b85a2a640 1 -- 172.21.15.64:0/2310076673 learned_addr learned my addr 172.21.15.64:0/2310076673 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:54.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.964+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345260111 0 0) 0x7f4b8006c870 con 0x7f4b8006c380 2024-03-29T11:30:54.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.964+0000 7f4b85229640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.964+0000 7f4b85229640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.964+0000 7f4b85229640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.964+0000 7f4b85229640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.964+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b6c003520 con 0x7f4b8006c380 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232581534 0 0) 0x7f4b6c003520 con 0x7f4b8006c380 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 >> v1:172.21.15.64:6790/0 conn(0x7f4b8006bfa0 legacy=0x7f4b80149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 >> v1:172.21.15.136:6789/0 conn(0x7f4b8014a050 legacy=0x7f4b80151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: _finish_hunting 0 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 1 monclient: found mon.noname-a 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80071f50 con 0x7f4b8006c380 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: _finish_auth 0 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: _check_auth_tickets 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:54.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b78001a10 con 0x7f4b8006c380 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.965+0000 7f4b85229640 10 monclient: dump: 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:54.960 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:54.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b8848f640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:30:54.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b8848f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:54.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b78001be0 con 0x7f4b8006c380 2024-03-29T11:30:54.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b85229640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:54.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b85229640 1 -- 172.21.15.64:0/2310076673 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b78002030 con 0x7f4b8006c380 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b8848f640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b8848f640 4 set_mon_vals no callback set 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b85229640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b85229640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.966+0000 7f4b85229640 10 monclient: dump: 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:54.963 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 10 monclient: shutdown 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 1 -- 172.21.15.64:0/2310076673 >> v1:172.21.15.64:6789/0 conn(0x7f4b8006c380 legacy=0x7f4b80071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 1 -- 172.21.15.64:0/2310076673 shutdown_connections 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 1 -- 172.21.15.64:0/2310076673 >> 172.21.15.64:0/2310076673 conn(0x7f4b80067340 msgr2=0x7f4b800b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 1 -- 172.21.15.64:0/2310076673 shutdown_connections 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.967+0000 7f4b8848f640 1 -- 172.21.15.64:0/2310076673 wait complete. 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.968+0000 7f4b8848f640 10 monclient: build_initial_monmap 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.968+0000 7f4b8848f640 10 monclient: monmap: 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:54.964 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.969+0000 7f4b8848f640 1 librados: starting msgr at 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.969+0000 7f4b8848f640 1 librados: starting objecter 2024-03-29T11:30:54.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.969+0000 7f4b8848f640 1 Processor -- start 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.972+0000 7f4b8848f640 1 -- start start 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.972+0000 7f4b8848f640 1 librados: setting wanted keys 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.972+0000 7f4b8848f640 1 librados: calling monclient init 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.972+0000 7f4b8848f640 10 monclient: init 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8006bfa0 addr v1:172.21.15.136:6789/0 2024-03-29T11:30:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient(hunting): picked mon.noname-c con 0x7f4b8006c380 addr v1:172.21.15.64:6790/0 2024-03-29T11:30:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8014a050 addr v1:172.21.15.64:6789/0 2024-03-29T11:30:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b800ab120 con 0x7f4b8014a050 2024-03-29T11:30:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b801dc2f0 con 0x7f4b8006bfa0 2024-03-29T11:30:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b801450b0 con 0x7f4b8006c380 2024-03-29T11:30:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.973+0000 7f4b8848f640 10 monclient(hunting): authenticate will time out at 1591.243619s 2024-03-29T11:30:54.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.978+0000 7f4b85a2a640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4b8006c380 0x7f4b800aaa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37729/0 (socket says 172.21.15.64:37729) 2024-03-29T11:30:54.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.978+0000 7f4b85a2a640 1 -- 172.21.15.64:0/518278422 learned_addr learned my addr 172.21.15.64:0/518278422 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:30:54.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 client.?.objecter ms_handle_connect 0x7f4b8006bfa0 2024-03-29T11:30:54.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 client.?.objecter ms_handle_connect 0x7f4b8006c380 2024-03-29T11:30:54.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126635166 0 0) 0x7f4b801450b0 con 0x7f4b8006c380 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b64004eb0 con 0x7f4b8006c380 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853135260 0 0) 0x7f4b801dc2f0 con 0x7f4b8006bfa0 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b801450b0 con 0x7f4b8006bfa0 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980069730 0 0) 0x7f4b64004eb0 con 0x7f4b8006c380 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b801dc2f0 con 0x7f4b8006c380 2024-03-29T11:30:54.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b78001ef0 con 0x7f4b8006c380 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:54.978 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.980+0000 7f4b76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4165898624 0 0) 0x7f4b801dc2f0 con 0x7f4b8006c380 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 >> v1:172.21.15.136:6789/0 conn(0x7f4b8006bfa0 legacy=0x7f4b800a9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 >> v1:172.21.15.64:6789/0 conn(0x7f4b8014a050 legacy=0x7f4b801dbbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 1 monclient: found mon.c 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80145240 con 0x7f4b8006c380 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.981+0000 7f4b76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.982+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b78004f70 con 0x7f4b8006c380 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.982+0000 7f4b76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:54.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.982+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b78005280 con 0x7f4b8006c380 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.982+0000 7f4b76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.982+0000 7f4b76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.982+0000 7f4b76ffd640 10 monclient: dump: 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:30:54.980 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:30:54.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 10 monclient: _renew_subs 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b801e7050 con 0x7f4b8006c380 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 10 monclient: _renew_subs 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b801e73c0 con 0x7f4b8006c380 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b8848f640 1 librados: init done 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.984+0000 7f4b84a28640 4 set_mon_vals no callback set 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.986+0000 7f4b8848f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.986+0000 7f4b8848f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.986+0000 7f4b8848f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:54.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.986+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b68005080 con 0x7f4b8006c380 2024-03-29T11:30:54.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.992+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4b7801b250 con 0x7f4b8006c380 2024-03-29T11:30:54.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f4b78084f50 con 0x7f4b8006c380 2024-03-29T11:30:54.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b8622b640 10 monclient: get_auth_request con 0x7f4b6406dd00 auth_method 0 2024-03-29T11:30:54.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b76ffd640 10 client.4499.objecter ms_dispatch 0x7f4b80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:54.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b76ffd640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:54.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b76ffd640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:54.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b76ffd640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:30:54.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.993+0000 7f4b76ffd640 10 client.4499.objecter ms_handle_connect 0x7f4b6406dd00 2024-03-29T11:30:54.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.996+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b78050830 con 0x7f4b8006c380 2024-03-29T11:30:54.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.996+0000 7f4b76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:54.996+0000 7f4b76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.320+0000 7f4b8848f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.320+0000 7f4b8848f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.320+0000 7f4b8848f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.320+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b68005330 con 0x7f4b8006c380 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.321+0000 7f4b76ffd640 1 -- 172.21.15.64:0/518278422 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f4b78052760 con 0x7f4b8006c380 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.321+0000 7f4b76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.321+0000 7f4b76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:55.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.324+0000 7f4b8848f640 10 librados: watch_flush enter 2024-03-29T11:30:55.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.324+0000 7f4b8848f640 10 librados: watch_flush exit 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.324+0000 7f4b8848f640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.324+0000 7f4b8848f640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 >> v1:172.21.15.136:6816/2312274743 conn(0x7f4b6406dd00 legacy=0x7f4b640701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 10 monclient: shutdown 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 >> v1:172.21.15.64:6790/0 conn(0x7f4b8006c380 legacy=0x7f4b800aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 shutdown_connections 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 >> 172.21.15.64:0/518278422 conn(0x7f4b80067340 msgr2=0x7f4b800b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 shutdown_connections 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 -- 172.21.15.64:0/518278422 wait complete. 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:30:55.325+0000 7f4b8848f640 1 librados: shutdown 2024-03-29T11:30:55.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:02.141 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:02.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.301+0000 7f1e21ef1640 10 monclient: get_monmap_and_config 2024-03-29T11:31:02.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.301+0000 7f1e21ef1640 10 monclient: build_initial_monmap 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.301+0000 7f1e21ef1640 10 monclient: monmap: 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:02.302794+0000 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:02.302794+0000 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:02.296 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:02.297 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:02.297 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:02.297 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.304+0000 7f1e21ef1640 1 Processor -- start 2024-03-29T11:31:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.304+0000 7f1e21ef1640 1 -- start start 2024-03-29T11:31:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.304+0000 7f1e21ef1640 10 monclient: init 2024-03-29T11:31:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f1e1c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f1e1c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f1e1c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c0aaef0 con 0x7f1e1c1457f0 2024-03-29T11:31:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c145bd0 con 0x7f1e1c0a9d70 2024-03-29T11:31:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c145eb0 con 0x7f1e1c13f650 2024-03-29T11:31:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.305+0000 7f1e21ef1640 10 monclient(hunting): authenticate will time out at 1598.575423s 2024-03-29T11:31:02.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.311+0000 7f1e1affd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1e1c13f650 0x7f1e1c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37359/0 (socket says 172.21.15.64:37359) 2024-03-29T11:31:02.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.311+0000 7f1e1affd640 1 -- 172.21.15.64:0/1156291839 learned_addr learned my addr 172.21.15.64:0/1156291839 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597800459 0 0) 0x7f1e1c145bd0 con 0x7f1e1c0a9d70 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e04003520 con 0x7f1e1c0a9d70 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778780353 0 0) 0x7f1e1c145eb0 con 0x7f1e1c13f650 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.313+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e1c145bd0 con 0x7f1e1c13f650 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70535413 0 0) 0x7f1e1c145bd0 con 0x7f1e1c13f650 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 >> v1:172.21.15.136:6789/0 conn(0x7f1e1c0a9d70 legacy=0x7f1e1c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 >> v1:172.21.15.64:6789/0 conn(0x7f1e1c1457f0 legacy=0x7f1e1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c146340 con 0x7f1e1c13f650 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:02.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1e10001670 con 0x7f1e1c13f650 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: dump: 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:02.310 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:02.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e10001890 con 0x7f1e1c13f650 2024-03-29T11:31:02.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:02.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 1 -- 172.21.15.64:0/1156291839 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1e10001c20 con 0x7f1e1c13f650 2024-03-29T11:31:02.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.314+0000 7f1e1a7fc640 10 monclient: dump: 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.315+0000 7f1e21ef1640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.315+0000 7f1e21ef1640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.315+0000 7f1e21ef1640 4 set_mon_vals no callback set 2024-03-29T11:31:02.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.315+0000 7f1e21ef1640 10 monclient: shutdown 2024-03-29T11:31:02.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.315+0000 7f1e21ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:02.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.315+0000 7f1e21ef1640 1 -- 172.21.15.64:0/1156291839 >> v1:172.21.15.64:6790/0 conn(0x7f1e1c13f650 legacy=0x7f1e1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.318+0000 7f1e21ef1640 1 -- 172.21.15.64:0/1156291839 shutdown_connections 2024-03-29T11:31:02.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.318+0000 7f1e21ef1640 1 -- 172.21.15.64:0/1156291839 >> 172.21.15.64:0/1156291839 conn(0x7f1e1c066a00 msgr2=0x7f1e1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.318+0000 7f1e21ef1640 1 -- 172.21.15.64:0/1156291839 shutdown_connections 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.318+0000 7f1e21ef1640 1 -- 172.21.15.64:0/1156291839 wait complete. 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.319+0000 7f1e21ef1640 10 monclient: build_initial_monmap 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.319+0000 7f1e21ef1640 10 monclient: monmap: 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:02.314 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 1 librados: starting msgr at 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 1 librados: starting objecter 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 1 Processor -- start 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 1 -- start start 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 1 librados: setting wanted keys 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 1 librados: calling monclient init 2024-03-29T11:31:02.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.320+0000 7f1e21ef1640 10 monclient: init 2024-03-29T11:31:02.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:02.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f1e1c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f1e1c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f1e1c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c1d81c0 con 0x7f1e1c1457f0 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c1d72b0 con 0x7f1e1c0a9d70 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c1d7590 con 0x7f1e1c13f650 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:02.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.321+0000 7f1e21ef1640 10 monclient(hunting): authenticate will time out at 1598.591456s 2024-03-29T11:31:02.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.326+0000 7f1e1b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1e1c0a9d70 0x7f1e1c1d7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:46533/0 (socket says 172.21.15.64:46533) 2024-03-29T11:31:02.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.326+0000 7f1e1b7fe640 1 -- 172.21.15.64:0/2250287652 learned_addr learned my addr 172.21.15.64:0/2250287652 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:02.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.326+0000 7f1e03fff640 10 client.?.objecter ms_handle_connect 0x7f1e1c0a9d70 2024-03-29T11:31:02.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.326+0000 7f1e03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.327+0000 7f1e03fff640 10 client.?.objecter ms_handle_connect 0x7f1e1c13f650 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.327+0000 7f1e03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251726007 0 0) 0x7f1e1c1d72b0 con 0x7f1e1c0a9d70 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e08004eb0 con 0x7f1e1c0a9d70 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254064099 0 0) 0x7f1e1c1d7590 con 0x7f1e1c13f650 2024-03-29T11:31:02.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e080051d0 con 0x7f1e1c13f650 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2734293635 0 0) 0x7f1e080051d0 con 0x7f1e1c13f650 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e08005640 con 0x7f1e1c13f650 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1e100015b0 con 0x7f1e1c13f650 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): dump: 2024-03-29T11:31:02.328 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:02.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1384811710 0 0) 0x7f1e08005640 con 0x7f1e1c13f650 2024-03-29T11:31:02.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 >> v1:172.21.15.136:6789/0 conn(0x7f1e1c0a9d70 legacy=0x7f1e1c1d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.328+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 >> v1:172.21.15.64:6789/0 conn(0x7f1e1c1457f0 legacy=0x7f1e1c1e1b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:02.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:02.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 1 monclient: found mon.c 2024-03-29T11:31:02.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:02.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:02.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c1d4d50 con 0x7f1e1c13f650 2024-03-29T11:31:02.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:02.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 10 monclient: _renew_subs 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c133b40 con 0x7f1e1c13f650 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 10 monclient: _renew_subs 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c1e3220 con 0x7f1e1c13f650 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e21ef1640 1 librados: init done 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e10004a90 con 0x7f1e1c13f650 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:02.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1e100015b0 con 0x7f1e1c13f650 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e03fff640 10 monclient: dump: 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:02.337 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.332+0000 7f1e197fa640 4 set_mon_vals no callback set 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.335+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f1e1001aaf0 con 0x7f1e1c13f650 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.336+0000 7f1e1b7fe640 10 monclient: get_auth_request con 0x7f1e0806e460 auth_method 0 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.336+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f1e10054450 con 0x7f1e1c13f650 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.336+0000 7f1e03fff640 10 client.4505.objecter ms_dispatch 0x7f1e1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.336+0000 7f1e03fff640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.336+0000 7f1e03fff640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:02.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.336+0000 7f1e03fff640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:31:02.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.337+0000 7f1e21ef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.338+0000 7f1e21ef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.338+0000 7f1e21ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:02.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.338+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dfc005080 con 0x7f1e1c13f650 2024-03-29T11:31:02.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.339+0000 7f1e03fff640 10 client.4505.objecter ms_handle_connect 0x7f1e0806e460 2024-03-29T11:31:02.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.348+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e100506f0 con 0x7f1e1c13f650 2024-03-29T11:31:02.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.348+0000 7f1e03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.348+0000 7f1e03fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.675+0000 7f1e21ef1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:02.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.675+0000 7f1e21ef1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:02.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.675+0000 7f1e21ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:02.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.675+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dfc005330 con 0x7f1e1c13f650 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.677+0000 7f1e03fff640 1 -- 172.21.15.64:0/2250287652 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f1e100506f0 con 0x7f1e1c13f650 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.677+0000 7f1e03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.677+0000 7f1e03fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:02.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 10 librados: watch_flush enter 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 10 librados: watch_flush exit 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 >> v1:172.21.15.136:6816/2312274743 conn(0x7f1e0806e460 legacy=0x7f1e08070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 10 monclient: shutdown 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 >> v1:172.21.15.64:6790/0 conn(0x7f1e1c13f650 legacy=0x7f1e1c133430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 shutdown_connections 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.681+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 >> 172.21.15.64:0/2250287652 conn(0x7f1e1c066a00 msgr2=0x7f1e1c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.682+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 shutdown_connections 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.682+0000 7f1e21ef1640 1 -- 172.21.15.64:0/2250287652 wait complete. 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:02.682+0000 7f1e21ef1640 1 librados: shutdown 2024-03-29T11:31:02.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:09.501 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:09.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.667+0000 7f4b2759e640 10 monclient: get_monmap_and_config 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.667+0000 7f4b2759e640 10 monclient: build_initial_monmap 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.667+0000 7f4b2759e640 10 monclient: monmap: 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:09.668847+0000 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:09.668847+0000 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.670+0000 7f4b2759e640 1 Processor -- start 2024-03-29T11:31:09.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.672+0000 7f4b2759e640 1 -- start start 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.672+0000 7f4b2759e640 10 monclient: init 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b20034850 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b2002a640 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b200327d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2002c820 con 0x7f4b200327d0 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2002a120 con 0x7f4b2002a640 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200c75b0 con 0x7f4b20034850 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.673+0000 7f4b2759e640 10 monclient(hunting): authenticate will time out at 1605.943266s 2024-03-29T11:31:09.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.677+0000 7f4b26d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4b200327d0 0x7f4b200cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37471/0 (socket says 172.21.15.64:37471) 2024-03-29T11:31:09.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.677+0000 7f4b26d9d640 1 -- 172.21.15.64:0/3251395251 learned_addr learned my addr 172.21.15.64:0/3251395251 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:09.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905564880 0 0) 0x7f4b2002c820 con 0x7f4b200327d0 2024-03-29T11:31:09.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:31:09.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b10003520 con 0x7f4b200327d0 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391645901 0 0) 0x7f4b2002a120 con 0x7f4b2002a640 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.679+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2002c820 con 0x7f4b2002a640 2024-03-29T11:31:09.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291789763 0 0) 0x7f4b10003520 con 0x7f4b200327d0 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 >> v1:172.21.15.64:6790/0 conn(0x7f4b20034850 legacy=0x7f4b200c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 >> v1:172.21.15.136:6789/0 conn(0x7f4b2002a640 legacy=0x7f4b200320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: _finish_hunting 0 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 monclient: found mon.noname-a 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b20033110 con 0x7f4b200327d0 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: _finish_auth 0 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: _check_auth_tickets 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b2804f730 con 0x7f4b200327d0 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: dump: 2024-03-29T11:31:09.675 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b280008d0 con 0x7f4b200327d0 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 1 -- 172.21.15.64:0/3251395251 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b280758b0 con 0x7f4b200327d0 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.680+0000 7f4b2559a640 10 monclient: dump: 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:09.677 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.682+0000 7f4b2759e640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:31:09.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.683+0000 7f4b2759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:09.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.683+0000 7f4b2759e640 4 set_mon_vals no callback set 2024-03-29T11:31:09.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.683+0000 7f4b2759e640 10 monclient: shutdown 2024-03-29T11:31:09.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.683+0000 7f4b2759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:09.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.683+0000 7f4b2759e640 1 -- 172.21.15.64:0/3251395251 >> v1:172.21.15.64:6789/0 conn(0x7f4b200327d0 legacy=0x7f4b200cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.685+0000 7f4b2759e640 1 -- 172.21.15.64:0/3251395251 shutdown_connections 2024-03-29T11:31:09.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.685+0000 7f4b2759e640 1 -- 172.21.15.64:0/3251395251 >> 172.21.15.64:0/3251395251 conn(0x7f4b200058f0 msgr2=0x7f4b20005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.686+0000 7f4b2759e640 1 -- 172.21.15.64:0/3251395251 shutdown_connections 2024-03-29T11:31:09.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.686+0000 7f4b2759e640 1 -- 172.21.15.64:0/3251395251 wait complete. 2024-03-29T11:31:09.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 10 monclient: build_initial_monmap 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 10 monclient: monmap: 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:09.684 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 1 librados: starting msgr at 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 1 librados: starting objecter 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 1 Processor -- start 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 1 -- start start 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 1 librados: setting wanted keys 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 1 librados: calling monclient init 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.687+0000 7f4b2759e640 10 monclient: init 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b2002a640 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b200327d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b20034850 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:09.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2002d0c0 con 0x7f4b200327d0 2024-03-29T11:31:09.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200c3a80 con 0x7f4b20034850 2024-03-29T11:31:09.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2002ded0 con 0x7f4b2002a640 2024-03-29T11:31:09.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:09.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.688+0000 7f4b2759e640 10 monclient(hunting): authenticate will time out at 1605.958748s 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.692+0000 7f4b25d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4b200327d0 0x7f4b200c1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37623/0 (socket says 172.21.15.64:37623) 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.692+0000 7f4b25d9b640 1 -- 172.21.15.64:0/3563110687 learned_addr learned my addr 172.21.15.64:0/3563110687 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.692+0000 7f4b16ffd640 10 client.?.objecter ms_handle_connect 0x7f4b200327d0 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.692+0000 7f4b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.693+0000 7f4b16ffd640 10 client.?.objecter ms_handle_connect 0x7f4b20034850 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.693+0000 7f4b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.693+0000 7f4b16ffd640 10 client.?.objecter ms_handle_connect 0x7f4b2002a640 2024-03-29T11:31:09.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:09.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871253153 0 0) 0x7f4b2002ded0 con 0x7f4b2002a640 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b08004eb0 con 0x7f4b2002a640 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142118352 0 0) 0x7f4b200c3a80 con 0x7f4b20034850 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.694+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2002ded0 con 0x7f4b20034850 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3977819041 0 0) 0x7f4b08004eb0 con 0x7f4b2002a640 2024-03-29T11:31:09.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b200c3a80 con 0x7f4b2002a640 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b18001810 con 0x7f4b2002a640 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:09.690 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:09.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294847989 0 0) 0x7f4b2002ded0 con 0x7f4b20034850 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b08004eb0 con 0x7f4b20034850 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b2804f400 con 0x7f4b20034850 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:09.692 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3988902205 0 0) 0x7f4b200c3a80 con 0x7f4b2002a640 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 >> v1:172.21.15.136:6789/0 conn(0x7f4b20034850 legacy=0x7f4b200c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 >> v1:172.21.15.64:6789/0 conn(0x7f4b200327d0 legacy=0x7f4b200c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 monclient: found mon.c 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200c2540 con 0x7f4b2002a640 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b2759e640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b2759e640 10 monclient: _renew_subs 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.695+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b200c2e40 con 0x7f4b2002a640 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.696+0000 7f4b2759e640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.696+0000 7f4b2759e640 10 monclient: _renew_subs 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.696+0000 7f4b2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.696+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b200bffc0 con 0x7f4b2002a640 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.696+0000 7f4b2759e640 1 librados: init done 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.698+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b18001970 con 0x7f4b2002a640 2024-03-29T11:31:09.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.698+0000 7f4b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:09.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.698+0000 7f4b17fff640 4 set_mon_vals no callback set 2024-03-29T11:31:09.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.699+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4b180037f0 con 0x7f4b2002a640 2024-03-29T11:31:09.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.699+0000 7f4b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.699+0000 7f4b16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:09.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.700+0000 7f4b16ffd640 10 monclient: dump: 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.700+0000 7f4b2759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.700+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4b18019880 con 0x7f4b2002a640 2024-03-29T11:31:09.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b2759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c005080 con 0x7f4b2002a640 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f4b18083940 con 0x7f4b2002a640 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b16ffd640 10 client.4511.objecter ms_dispatch 0x7f4b28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b16ffd640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b16ffd640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.701+0000 7f4b16ffd640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:31:09.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.702+0000 7f4b25d9b640 10 monclient: get_auth_request con 0x7f4b0806ddd0 auth_method 0 2024-03-29T11:31:09.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.706+0000 7f4b16ffd640 10 client.4511.objecter ms_handle_connect 0x7f4b0806ddd0 2024-03-29T11:31:09.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.710+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b1804f0d0 con 0x7f4b2002a640 2024-03-29T11:31:09.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.710+0000 7f4b16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:09.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:09.710+0000 7f4b16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.034+0000 7f4b2759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.034+0000 7f4b2759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.034+0000 7f4b2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.034+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b0c005330 con 0x7f4b2002a640 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.039+0000 7f4b16ffd640 1 -- 172.21.15.64:0/3563110687 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f4b180511d0 con 0x7f4b2002a640 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.039+0000 7f4b16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.039+0000 7f4b16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 10 librados: watch_flush enter 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 10 librados: watch_flush exit 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 >> v1:172.21.15.136:6816/2312274743 conn(0x7f4b0806ddd0 legacy=0x7f4b08070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 10 monclient: shutdown 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:10.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.043+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 >> v1:172.21.15.64:6790/0 conn(0x7f4b2002a640 legacy=0x7f4b2002d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.044+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 shutdown_connections 2024-03-29T11:31:10.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.044+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 >> 172.21.15.64:0/3563110687 conn(0x7f4b200058f0 msgr2=0x7f4b200043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.044+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 shutdown_connections 2024-03-29T11:31:10.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.045+0000 7f4b2759e640 1 -- 172.21.15.64:0/3563110687 wait complete. 2024-03-29T11:31:10.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:10.046+0000 7f4b2759e640 1 librados: shutdown 2024-03-29T11:31:10.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:16.866 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:17.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.033+0000 7fe85d919640 10 monclient: get_monmap_and_config 2024-03-29T11:31:17.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.033+0000 7fe85d919640 10 monclient: build_initial_monmap 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.033+0000 7fe85d919640 10 monclient: monmap: 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:17.035273+0000 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:17.035273+0000 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:17.029 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.035+0000 7fe85d919640 1 Processor -- start 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.035+0000 7fe85d919640 1 -- start start 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.035+0000 7fe85d919640 10 monclient: init 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.036+0000 7fe85d919640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.036+0000 7fe85d919640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.036+0000 7fe85d919640 10 monclient(hunting): picked mon.noname-c con 0x7fe8580a9750 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.036+0000 7fe85d919640 10 monclient(hunting): picked mon.noname-a con 0x7fe8580adca0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.036+0000 7fe85d919640 10 monclient(hunting): picked mon.noname-b con 0x7fe858146a80 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.036+0000 7fe85d919640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8580abdc0 con 0x7fe8580adca0 2024-03-29T11:31:17.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.037+0000 7fe85d919640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8580b0070 con 0x7fe8580a9750 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.037+0000 7fe85d919640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8580af880 con 0x7fe858146a80 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.037+0000 7fe85d919640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.037+0000 7fe85d919640 10 monclient(hunting): authenticate will time out at 1613.307895s 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.040+0000 7fe85c917640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe8580a9750 0x7fe858143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:55737/0 (socket says 172.21.15.64:55737) 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.040+0000 7fe85c917640 1 -- 172.21.15.64:0/1284327046 learned_addr learned my addr 172.21.15.64:0/1284327046 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.041+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125271671 0 0) 0x7fe8580b0070 con 0x7fe8580a9750 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.041+0000 7fe8577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.041+0000 7fe8577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.041+0000 7fe8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.041+0000 7fe8577fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:31:17.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.041+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe840003520 con 0x7fe8580a9750 2024-03-29T11:31:17.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58543530 0 0) 0x7fe8580af880 con 0x7fe858146a80 2024-03-29T11:31:17.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:31:17.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8580b0070 con 0x7fe858146a80 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3801188954 0 0) 0x7fe840003520 con 0x7fe8580a9750 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 >> v1:172.21.15.64:6790/0 conn(0x7fe858146a80 legacy=0x7fe858151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.042+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 >> v1:172.21.15.64:6789/0 conn(0x7fe8580adca0 legacy=0x7fe858146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 1 monclient: found mon.noname-c 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8580ac030 con 0x7fe8580a9750 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe84c001610 con 0x7fe8580a9750 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: dump: 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe85d919640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe85d919640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe84c001770 con 0x7fe8580a9750 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:17.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 1 -- 172.21.15.64:0/1284327046 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe84c001bc0 con 0x7fe8580a9750 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe8577fe640 10 monclient: dump: 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe85d919640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:17.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.043+0000 7fe85d919640 4 set_mon_vals no callback set 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.044+0000 7fe85d919640 10 monclient: shutdown 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.044+0000 7fe85d919640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.044+0000 7fe85d919640 1 -- 172.21.15.64:0/1284327046 >> v1:172.21.15.136:6789/0 conn(0x7fe8580a9750 legacy=0x7fe858143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.045+0000 7fe85d919640 1 -- 172.21.15.64:0/1284327046 shutdown_connections 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.045+0000 7fe85d919640 1 -- 172.21.15.64:0/1284327046 >> 172.21.15.64:0/1284327046 conn(0x7fe8580673e0 msgr2=0x7fe8580b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.045+0000 7fe85d919640 1 -- 172.21.15.64:0/1284327046 shutdown_connections 2024-03-29T11:31:17.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.045+0000 7fe85d919640 1 -- 172.21.15.64:0/1284327046 wait complete. 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.046+0000 7fe85d919640 10 monclient: build_initial_monmap 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.046+0000 7fe85d919640 10 monclient: monmap: 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:17.046 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.047+0000 7fe85d919640 1 librados: starting msgr at 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.047+0000 7fe85d919640 1 librados: starting objecter 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.047+0000 7fe85d919640 1 Processor -- start 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.048+0000 7fe85d919640 1 -- start start 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.048+0000 7fe85d919640 1 librados: setting wanted keys 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.048+0000 7fe85d919640 1 librados: calling monclient init 2024-03-29T11:31:17.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.048+0000 7fe85d919640 10 monclient: init 2024-03-29T11:31:17.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:17.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:17.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient(hunting): picked mon.noname-a con 0x7fe8580a9750 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:17.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient(hunting): picked mon.noname-c con 0x7fe8580adca0 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient(hunting): picked mon.noname-b con 0x7fe858146a80 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8581dc260 con 0x7fe8580a9750 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8581db630 con 0x7fe858146a80 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8581cf580 con 0x7fe8580adca0 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.049+0000 7fe85d919640 10 monclient(hunting): authenticate will time out at 1613.320414s 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.053+0000 7fe857fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe8580adca0 0x7fe8581d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49130/0 (socket says 172.21.15.64:49130) 2024-03-29T11:31:17.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.053+0000 7fe857fff640 1 -- 172.21.15.64:0/4048897063 learned_addr learned my addr 172.21.15.64:0/4048897063 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 client.?.objecter ms_handle_connect 0x7fe8580adca0 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 client.?.objecter ms_handle_connect 0x7fe858146a80 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 client.?.objecter ms_handle_connect 0x7fe8580a9750 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186413108 0 0) 0x7fe8581db630 con 0x7fe858146a80 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:31:17.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.054+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe844004eb0 con 0x7fe858146a80 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219470091 0 0) 0x7fe8581cf580 con 0x7fe8580adca0 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8581db630 con 0x7fe8580adca0 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936654575 0 0) 0x7fe844004eb0 con 0x7fe858146a80 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8581cf580 con 0x7fe858146a80 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe850001b10 con 0x7fe858146a80 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): dump: 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.051 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:17.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4113176526 0 0) 0x7fe8581cf580 con 0x7fe858146a80 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 >> v1:172.21.15.64:6790/0 conn(0x7fe8580adca0 legacy=0x7fe8581d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.055+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 >> v1:172.21.15.64:6789/0 conn(0x7fe8580a9750 legacy=0x7fe8581dbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 1 monclient: found mon.b 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:17.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8581cf710 con 0x7fe858146a80 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe850001fc0 con 0x7fe858146a80 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe8557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.056+0000 7fe856ffd640 4 set_mon_vals no callback set 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 10 monclient: _renew_subs 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8581d08a0 con 0x7fe858146a80 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 10 monclient: _renew_subs 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:17.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8581d0bf0 con 0x7fe858146a80 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe85d919640 1 librados: init done 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe850003b30 con 0x7fe858146a80 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe8557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe8557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.057+0000 7fe8557fa640 10 monclient: dump: 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:17.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.058+0000 7fe85d919640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:17.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.059+0000 7fe85d919640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:17.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.059+0000 7fe85d919640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:17.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.059+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8581db7c0 con 0x7fe858146a80 2024-03-29T11:31:17.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.062+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fe850019b00 con 0x7fe858146a80 2024-03-29T11:31:17.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.063+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fe850083a50 con 0x7fe858146a80 2024-03-29T11:31:17.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.063+0000 7fe8557fa640 10 client.4510.objecter ms_dispatch 0x7fe858000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:17.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.063+0000 7fe8557fa640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:17.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.063+0000 7fe8557fa640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:17.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.063+0000 7fe8557fa640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:31:17.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.063+0000 7fe85c917640 10 monclient: get_auth_request con 0x7fe84406ddc0 auth_method 0 2024-03-29T11:31:17.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.064+0000 7fe8557fa640 10 client.4510.objecter ms_handle_connect 0x7fe84406ddc0 2024-03-29T11:31:17.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.067+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe85004f1e0 con 0x7fe858146a80 2024-03-29T11:31:17.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.067+0000 7fe8557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:17.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.067+0000 7fe8557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:17.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.418+0000 7fe85d919640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.418+0000 7fe85d919640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.418+0000 7fe85d919640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:17.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.418+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8581db460 con 0x7fe858146a80 2024-03-29T11:31:17.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.420+0000 7fe8557fa640 1 -- 172.21.15.64:0/4048897063 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fe8500512e0 con 0x7fe858146a80 2024-03-29T11:31:17.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.420+0000 7fe8557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.420+0000 7fe8557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:17.415 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:17.415 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:17.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.424+0000 7fe85d919640 10 librados: watch_flush enter 2024-03-29T11:31:17.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.424+0000 7fe85d919640 10 librados: watch_flush exit 2024-03-29T11:31:17.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.424+0000 7fe85d919640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:31:17.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.424+0000 7fe85d919640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:31:17.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.425+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 >> v1:172.21.15.136:6816/2312274743 conn(0x7fe84406ddc0 legacy=0x7fe844070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.425+0000 7fe85d919640 10 monclient: shutdown 2024-03-29T11:31:17.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.425+0000 7fe85d919640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:17.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.425+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 >> v1:172.21.15.136:6789/0 conn(0x7fe858146a80 legacy=0x7fe8581e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.425+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 shutdown_connections 2024-03-29T11:31:17.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.425+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 >> 172.21.15.64:0/4048897063 conn(0x7fe8580673e0 msgr2=0x7fe858069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.428+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 shutdown_connections 2024-03-29T11:31:17.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.428+0000 7fe85d919640 1 -- 172.21.15.64:0/4048897063 wait complete. 2024-03-29T11:31:17.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:17.428+0000 7fe85d919640 1 librados: shutdown 2024-03-29T11:31:17.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:24.247 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:24.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.399+0000 7f4745a2d640 10 monclient: get_monmap_and_config 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.399+0000 7f4745a2d640 10 monclient: build_initial_monmap 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.399+0000 7f4745a2d640 10 monclient: monmap: 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:24.400605+0000 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:24.400605+0000 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:24.394 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:24.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.401+0000 7f4745a2d640 1 Processor -- start 2024-03-29T11:31:24.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.401+0000 7f4745a2d640 1 -- start start 2024-03-29T11:31:24.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.401+0000 7f4745a2d640 10 monclient: init 2024-03-29T11:31:24.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.402+0000 7f4745a2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:24.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.402+0000 7f4745a2d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:24.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.402+0000 7f4745a2d640 10 monclient(hunting): picked mon.noname-b con 0x7f47400b4820 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:24.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.402+0000 7f4745a2d640 10 monclient(hunting): picked mon.noname-a con 0x7f47400a9ce0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:24.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f473ffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f47400a9ce0 0x7f4740145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:49342/0 (socket says 172.21.15.64:49342) 2024-03-29T11:31:24.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f474522c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f47400b3cd0 0x7f474014d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:35200/0 (socket says 172.21.15.64:35200) 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f4745a2d640 10 monclient(hunting): picked mon.noname-c con 0x7f47400b3cd0 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f4745a2d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47400b4180 con 0x7f47400a9ce0 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f4745a2d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47400aaf60 con 0x7f47400b3cd0 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f4745a2d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474013c200 con 0x7f47400b4820 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f4745a2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f4745a2d640 10 monclient(hunting): authenticate will time out at 1620.673390s 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.403+0000 7f473ffff640 1 -- 172.21.15.64:0/2945855575 learned_addr learned my addr 172.21.15.64:0/2945855575 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995047358 0 0) 0x7f47400aaf60 con 0x7f47400b3cd0 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4730003520 con 0x7f47400b3cd0 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632641746 0 0) 0x7f472c001110 con 0x7f47400a9ce0 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47400aaf60 con 0x7f47400a9ce0 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172235224 0 0) 0x7f474013c200 con 0x7f47400b4820 2024-03-29T11:31:24.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.404+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f472c001110 con 0x7f47400b4820 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3422842475 0 0) 0x7f47400aaf60 con 0x7f47400a9ce0 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 >> v1:172.21.15.64:6790/0 conn(0x7f47400b4820 legacy=0x7f474013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 >> v1:172.21.15.136:6789/0 conn(0x7f47400b3cd0 legacy=0x7f474014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740145e80 con 0x7f47400a9ce0 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:24.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f472c001660 con 0x7f47400a9ce0 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f473f7fe640 10 monclient: dump: 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:24.402 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:24.403 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:24.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f4745a2d640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:31:24.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.405+0000 7f4745a2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:24.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f472c0017c0 con 0x7f47400a9ce0 2024-03-29T11:31:24.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f473f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f473f7fe640 1 -- 172.21.15.64:0/2945855575 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f472c001c10 con 0x7f47400a9ce0 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f473f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f473f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f473f7fe640 10 monclient: dump: 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.406 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f4745a2d640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:24.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.406+0000 7f4745a2d640 4 set_mon_vals no callback set 2024-03-29T11:31:24.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 10 monclient: shutdown 2024-03-29T11:31:24.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:24.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 1 -- 172.21.15.64:0/2945855575 >> v1:172.21.15.64:6789/0 conn(0x7f47400a9ce0 legacy=0x7f4740145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 1 -- 172.21.15.64:0/2945855575 shutdown_connections 2024-03-29T11:31:24.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 1 -- 172.21.15.64:0/2945855575 >> 172.21.15.64:0/2945855575 conn(0x7f474006d070 msgr2=0x7f4740066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:24.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 1 -- 172.21.15.64:0/2945855575 shutdown_connections 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.407+0000 7f4745a2d640 1 -- 172.21.15.64:0/2945855575 wait complete. 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.408+0000 7f4745a2d640 10 monclient: build_initial_monmap 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.408+0000 7f4745a2d640 10 monclient: monmap: 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:24.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 1 librados: starting msgr at 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 1 librados: starting objecter 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 1 Processor -- start 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 1 -- start start 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 1 librados: setting wanted keys 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 1 librados: calling monclient init 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.409+0000 7f4745a2d640 10 monclient: init 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient(hunting): picked mon.noname-c con 0x7f47400a9ce0 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient(hunting): picked mon.noname-b con 0x7f47400b3cd0 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient(hunting): picked mon.noname-a con 0x7f47400b4820 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4740070770 con 0x7f47400b4820 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474006e970 con 0x7f47400b3cd0 2024-03-29T11:31:24.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4740071f80 con 0x7f47400a9ce0 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4745a2d640 10 monclient(hunting): authenticate will time out at 1620.680644s 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.410+0000 7f4744a2b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f47400a9ce0 0x7f4740071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37096/0 (socket says 172.21.15.64:37096) 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f4744a2b640 1 -- 172.21.15.64:0/808955440 learned_addr learned my addr 172.21.15.64:0/808955440 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 client.?.objecter ms_handle_connect 0x7f47400b3cd0 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 client.?.objecter ms_handle_connect 0x7f47400a9ce0 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590471561 0 0) 0x7f474006e970 con 0x7f47400b3cd0 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:31:24.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.411+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4724004eb0 con 0x7f47400b3cd0 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 client.?.objecter ms_handle_connect 0x7f47400b4820 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727833462 0 0) 0x7f4740071f80 con 0x7f47400a9ce0 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f474006e970 con 0x7f47400a9ce0 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1649609498 0 0) 0x7f474006e970 con 0x7f47400a9ce0 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4740071f80 con 0x7f47400a9ce0 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4734001860 con 0x7f47400a9ce0 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.412+0000 7f473d7fa640 10 monclient(hunting): dump: 2024-03-29T11:31:24.418 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:24.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1536339916 0 0) 0x7f4724004eb0 con 0x7f47400b3cd0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f474006e970 con 0x7f47400b3cd0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 143139688 0 0) 0x7f4740071f80 con 0x7f47400a9ce0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 >> v1:172.21.15.136:6789/0 conn(0x7f47400b3cd0 legacy=0x7f474006ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 >> v1:172.21.15.64:6789/0 conn(0x7f47400b4820 legacy=0x7f47401e6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 monclient: found mon.c 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740072110 con 0x7f47400a9ce0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:24.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f473d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 10 monclient: _renew_subs 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474006f5a0 con 0x7f47400a9ce0 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 10 monclient: _renew_subs 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47401dc200 con 0x7f47400a9ce0 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.413+0000 7f4745a2d640 1 librados: init done 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47340019c0 con 0x7f47400a9ce0 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:24.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f47340037b0 con 0x7f47400a9ce0 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473e7fc640 4 set_mon_vals no callback set 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f473d7fa640 10 monclient: dump: 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:24.422 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.415+0000 7f4745a2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.416+0000 7f4745a2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.416+0000 7f4745a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.416+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4720005080 con 0x7f47400a9ce0 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.417+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4734003910 con 0x7f47400a9ce0 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.418+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f4734084870 con 0x7f47400a9ce0 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.418+0000 7f473d7fa640 10 client.4523.objecter ms_dispatch 0x7f4740000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.418+0000 7f473d7fa640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.418+0000 7f473d7fa640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.419+0000 7f473d7fa640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.419+0000 7f473ffff640 10 monclient: get_auth_request con 0x7f472406ddb0 auth_method 0 2024-03-29T11:31:24.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.419+0000 7f473d7fa640 10 client.4523.objecter ms_handle_connect 0x7f472406ddb0 2024-03-29T11:31:24.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.432+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4734050150 con 0x7f47400a9ce0 2024-03-29T11:31:24.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.432+0000 7f473d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.432+0000 7f473d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:24.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.829+0000 7f4745a2d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:24.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.829+0000 7f4745a2d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:24.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.829+0000 7f4745a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:24.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.829+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4720005330 con 0x7f47400a9ce0 2024-03-29T11:31:24.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.833+0000 7f473d7fa640 1 -- 172.21.15.64:0/808955440 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f4734052080 con 0x7f47400a9ce0 2024-03-29T11:31:24.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.833+0000 7f473d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:24.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.833+0000 7f473d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:24.827 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:24.827 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:24.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.837+0000 7f4745a2d640 10 librados: watch_flush enter 2024-03-29T11:31:24.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.837+0000 7f4745a2d640 10 librados: watch_flush exit 2024-03-29T11:31:24.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.837+0000 7f4745a2d640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:31:24.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.837+0000 7f4745a2d640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:31:24.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 >> v1:172.21.15.136:6816/2312274743 conn(0x7f472406ddb0 legacy=0x7f4724070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 10 monclient: shutdown 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 >> v1:172.21.15.64:6790/0 conn(0x7f47400a9ce0 legacy=0x7f4740071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 shutdown_connections 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 >> 172.21.15.64:0/808955440 conn(0x7f474006d070 msgr2=0x7f47400b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 shutdown_connections 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.838+0000 7f4745a2d640 1 -- 172.21.15.64:0/808955440 wait complete. 2024-03-29T11:31:24.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:24.839+0000 7f4745a2d640 1 librados: shutdown 2024-03-29T11:31:24.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:31.656 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:31.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.814+0000 7fd55188b640 10 monclient: get_monmap_and_config 2024-03-29T11:31:31.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.814+0000 7fd55188b640 10 monclient: build_initial_monmap 2024-03-29T11:31:31.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.815+0000 7fd55188b640 10 monclient: monmap: 2024-03-29T11:31:31.810 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:31.815952+0000 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:31.815952+0000 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:31.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.816+0000 7fd55188b640 1 Processor -- start 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.816+0000 7fd55188b640 1 -- start start 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.816+0000 7fd55188b640 10 monclient: init 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient(hunting): picked mon.noname-b con 0x7fd54c0a9750 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient(hunting): picked mon.noname-c con 0x7fd54c0ae220 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient(hunting): picked mon.noname-a con 0x7fd54c13b9d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c142490 con 0x7fd54c13b9d0 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0ac130 con 0x7fd54c0ae220 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0ac410 con 0x7fd54c0a9750 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:31.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.817+0000 7fd55188b640 10 monclient(hunting): authenticate will time out at 1628.087219s 2024-03-29T11:31:31.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.820+0000 7fd550889640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd54c0a9750 0x7fd54c141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35739/0 (socket says 172.21.15.64:35739) 2024-03-29T11:31:31.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.820+0000 7fd550889640 1 -- 172.21.15.64:0/3003625892 learned_addr learned my addr 172.21.15.64:0/3003625892 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409699347 0 0) 0x7fd54c142490 con 0x7fd54c13b9d0 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534003520 con 0x7fd54c13b9d0 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978757065 0 0) 0x7fd54c0ac410 con 0x7fd54c0a9750 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:31:31.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.821+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c142490 con 0x7fd54c0a9750 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037222350 0 0) 0x7fd54c142490 con 0x7fd54c0a9750 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 >> v1:172.21.15.136:6789/0 conn(0x7fd54c0ae220 legacy=0x7fd54c13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 >> v1:172.21.15.64:6789/0 conn(0x7fd54c13b9d0 legacy=0x7fd54c149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:31:31.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0afa40 con 0x7fd54c0a9750 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd53c001670 con 0x7fd54c0a9750 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: dump: 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:31.820 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd53c001890 con 0x7fd54c0a9750 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 1 -- 172.21.15.64:0/3003625892 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd53c001ba0 con 0x7fd54c0a9750 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.822+0000 7fd54b7fe640 10 monclient: dump: 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:31.821 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:31.822 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:31.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.823+0000 7fd55188b640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:31:31.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.823+0000 7fd55188b640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:31.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.823+0000 7fd55188b640 4 set_mon_vals no callback set 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 10 monclient: shutdown 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 1 -- 172.21.15.64:0/3003625892 >> v1:172.21.15.64:6790/0 conn(0x7fd54c0a9750 legacy=0x7fd54c141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 1 -- 172.21.15.64:0/3003625892 shutdown_connections 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 1 -- 172.21.15.64:0/3003625892 >> 172.21.15.64:0/3003625892 conn(0x7fd54c0673e0 msgr2=0x7fd54c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 1 -- 172.21.15.64:0/3003625892 shutdown_connections 2024-03-29T11:31:31.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.826+0000 7fd55188b640 1 -- 172.21.15.64:0/3003625892 wait complete. 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.827+0000 7fd55188b640 10 monclient: build_initial_monmap 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.827+0000 7fd55188b640 10 monclient: monmap: 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:31.824 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 1 librados: starting msgr at 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 1 librados: starting objecter 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 1 Processor -- start 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 1 -- start start 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 1 librados: setting wanted keys 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 1 librados: calling monclient init 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.828+0000 7fd55188b640 10 monclient: init 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient(hunting): picked mon.noname-a con 0x7fd54c0a9750 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:31.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient(hunting): picked mon.noname-b con 0x7fd54c0ae220 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient(hunting): picked mon.noname-c con 0x7fd54c13b9d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c06fc00 con 0x7fd54c0a9750 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c06e5b0 con 0x7fd54c0ae220 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c134380 con 0x7fd54c13b9d0 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.829+0000 7fd55188b640 10 monclient(hunting): authenticate will time out at 1628.099467s 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.834+0000 7fd54bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd54c0ae220 0x7fd54c133560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:41359/0 (socket says 172.21.15.64:41359) 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.834+0000 7fd54bfff640 1 -- 172.21.15.64:0/1789200701 learned_addr learned my addr 172.21.15.64:0/1789200701 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 client.?.objecter ms_handle_connect 0x7fd54c0ae220 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149384961 0 0) 0x7fd54c06e5b0 con 0x7fd54c0ae220 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:31.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:31.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:31:31.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd538004eb0 con 0x7fd54c0ae220 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3662597813 0 0) 0x7fd538004eb0 con 0x7fd54c0ae220 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd54c06e5b0 con 0x7fd54c0ae220 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd544001d50 con 0x7fd54c0ae220 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): dump: 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:31.831 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2101808364 0 0) 0x7fd54c06e5b0 con 0x7fd54c0ae220 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.835+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 >> v1:172.21.15.64:6790/0 conn(0x7fd54c13b9d0 legacy=0x7fd54c133c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 >> v1:172.21.15.64:6789/0 conn(0x7fd54c0a9750 legacy=0x7fd54c070290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 1 monclient: found mon.b 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c06fe70 con 0x7fd54c0ae220 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:31.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd5497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd55188b640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd55188b640 10 monclient: _renew_subs 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd55188b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.836+0000 7fd55188b640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c0709a0 con 0x7fd54c0ae220 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.837+0000 7fd55188b640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.837+0000 7fd55188b640 10 monclient: _renew_subs 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.837+0000 7fd55188b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.837+0000 7fd55188b640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c1da3e0 con 0x7fd54c0ae220 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.837+0000 7fd55188b640 1 librados: init done 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.838+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd544002050 con 0x7fd54c0ae220 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.838+0000 7fd5497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.838+0000 7fd54a7fc640 4 set_mon_vals no callback set 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.839+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd544003de0 con 0x7fd54c0ae220 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.839+0000 7fd5497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.839+0000 7fd5497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.840+0000 7fd5497fa640 10 monclient: dump: 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:31.836 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.840+0000 7fd532ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.841+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fd544004000 con 0x7fd54c0ae220 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.842+0000 7fd550889640 10 monclient: get_auth_request con 0x7fd53806dca0 auth_method 0 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.842+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fd544084f20 con 0x7fd54c0ae220 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.842+0000 7fd5497fa640 10 client.4522.objecter ms_dispatch 0x7fd54c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.842+0000 7fd5497fa640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.842+0000 7fd5497fa640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.843+0000 7fd532ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.843+0000 7fd532ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.843+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c005080 con 0x7fd54c0ae220 2024-03-29T11:31:31.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.844+0000 7fd5497fa640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:31:31.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.844+0000 7fd5497fa640 10 client.4522.objecter ms_handle_connect 0x7fd53806dca0 2024-03-29T11:31:31.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.851+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd544050880 con 0x7fd54c0ae220 2024-03-29T11:31:31.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.851+0000 7fd5497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:31.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:31.851+0000 7fd5497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:32.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.220+0000 7fd532ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:32.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.220+0000 7fd532ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:32.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.220+0000 7fd532ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:32.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.220+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd52c005330 con 0x7fd54c0ae220 2024-03-29T11:31:32.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.222+0000 7fd5497fa640 1 -- 172.21.15.64:0/1789200701 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fd5440527b0 con 0x7fd54c0ae220 2024-03-29T11:31:32.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.222+0000 7fd5497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:32.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.222+0000 7fd5497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:32.216 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:32.217 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 10 librados: watch_flush enter 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 10 librados: watch_flush exit 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 >> v1:172.21.15.136:6816/2312274743 conn(0x7fd53806dca0 legacy=0x7fd538070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 10 monclient: shutdown 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 >> v1:172.21.15.136:6789/0 conn(0x7fd54c0ae220 legacy=0x7fd54c133560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 shutdown_connections 2024-03-29T11:31:32.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.225+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 >> 172.21.15.64:0/1789200701 conn(0x7fd54c0673e0 msgr2=0x7fd54c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:32.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.226+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 shutdown_connections 2024-03-29T11:31:32.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.226+0000 7fd532ffd640 1 -- 172.21.15.64:0/1789200701 wait complete. 2024-03-29T11:31:32.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:32.226+0000 7fd532ffd640 1 librados: shutdown 2024-03-29T11:31:32.234 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:39.042 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:39.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.199+0000 7f81d7c0d640 10 monclient: get_monmap_and_config 2024-03-29T11:31:39.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.199+0000 7f81d7c0d640 10 monclient: build_initial_monmap 2024-03-29T11:31:39.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.199+0000 7f81d7c0d640 10 monclient: monmap: 2024-03-29T11:31:39.196 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:39.196 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:39.196 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:39.200752+0000 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:39.200752+0000 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.200+0000 7f81d7c0d640 1 Processor -- start 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.200+0000 7f81d7c0d640 1 -- start start 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.200+0000 7f81d7c0d640 10 monclient: init 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.201+0000 7f81d7c0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.201+0000 7f81d7c0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:31:39.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.201+0000 7f81d7c0d640 10 monclient(hunting): picked mon.noname-c con 0x7f81d00ad6c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.202+0000 7f81d7c0d640 10 monclient(hunting): picked mon.noname-a con 0x7f81d00af6c0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d59a9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f81d00ad6c0 0x7f81d014f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:34174/0 (socket says 172.21.15.64:34174) 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d7c0d640 10 monclient(hunting): picked mon.noname-b con 0x7f81d0153090 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d59a9640 1 -- 172.21.15.64:0/3204773774 learned_addr learned my addr 172.21.15.64:0/3204773774 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d00a94e0 con 0x7f81d00af6c0 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d00af3a0 con 0x7f81d00ad6c0 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d00afb60 con 0x7f81d0153090 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d7c0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.203+0000 7f81d7c0d640 10 monclient(hunting): authenticate will time out at 1635.473065s 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.204+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261680822 0 0) 0x7f81d00a94e0 con 0x7f81d00af6c0 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81bc003520 con 0x7f81d00af6c0 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046119597 0 0) 0x7f81d00af3a0 con 0x7f81d00ad6c0 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d00a94e0 con 0x7f81d00ad6c0 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1263613278 0 0) 0x7f81bc003520 con 0x7f81d00af6c0 2024-03-29T11:31:39.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.205+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 >> v1:172.21.15.64:6790/0 conn(0x7f81d0153090 legacy=0x7f81d015a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.206+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 >> v1:172.21.15.136:6789/0 conn(0x7f81d00ad6c0 legacy=0x7f81d014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.206+0000 7f81d49a7640 10 monclient: _finish_hunting 0 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 1 monclient: found mon.noname-a 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00aab80 con 0x7f81d00af6c0 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 10 monclient: _finish_auth 0 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 10 monclient: _check_auth_tickets 2024-03-29T11:31:39.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f81c00015c0 con 0x7f81d00af6c0 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d49a7640 10 monclient: dump: 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:39.205 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d7c0d640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.207+0000 7f81d7c0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c00017e0 con 0x7f81d00af6c0 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d49a7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d49a7640 1 -- 172.21.15.64:0/3204773774 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f81c0001b70 con 0x7f81d00af6c0 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d49a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d49a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d49a7640 10 monclient: dump: 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.206 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d7c0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.208+0000 7f81d7c0d640 4 set_mon_vals no callback set 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 10 monclient: shutdown 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 >> v1:172.21.15.64:6789/0 conn(0x7f81d00af6c0 legacy=0x7f81d0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 shutdown_connections 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 >> 172.21.15.64:0/3204773774 conn(0x7f81d00b35e0 msgr2=0x7f81d00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 shutdown_connections 2024-03-29T11:31:39.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.209+0000 7f81d7c0d640 1 -- 172.21.15.64:0/3204773774 wait complete. 2024-03-29T11:31:39.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.210+0000 7f81d7c0d640 10 monclient: build_initial_monmap 2024-03-29T11:31:39.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.210+0000 7f81d7c0d640 10 monclient: monmap: 2024-03-29T11:31:39.208 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 1 librados: starting msgr at 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 1 librados: starting objecter 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 1 Processor -- start 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 1 -- start start 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 1 librados: setting wanted keys 2024-03-29T11:31:39.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 1 librados: calling monclient init 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.211+0000 7f81d7c0d640 10 monclient: init 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.212+0000 7f81d7c0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.212+0000 7f81d7c0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 10 monclient(hunting): picked mon.noname-c con 0x7f81d00ad6c0 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 10 monclient(hunting): picked mon.noname-b con 0x7f81d00af6c0 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 10 monclient(hunting): picked mon.noname-a con 0x7f81d0153090 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d00716d0 con 0x7f81d0153090 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d0071bf0 con 0x7f81d00af6c0 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d0070630 con 0x7f81d00ad6c0 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:39.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.214+0000 7f81d7c0d640 10 monclient(hunting): authenticate will time out at 1635.484272s 2024-03-29T11:31:39.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.215+0000 7f81d51a8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f81d00af6c0 0x7f81d01e41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:34760/0 (socket says 172.21.15.64:34760) 2024-03-29T11:31:39.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.215+0000 7f81d51a8640 1 -- 172.21.15.64:0/1666931235 learned_addr learned my addr 172.21.15.64:0/1666931235 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.216+0000 7f81c67fc640 10 client.?.objecter ms_handle_connect 0x7f81d00af6c0 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.216+0000 7f81c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.216+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344189231 0 0) 0x7f81d0071bf0 con 0x7f81d00af6c0 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.216+0000 7f81c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:31:39.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b4004eb0 con 0x7f81d00af6c0 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3666941797 0 0) 0x7f81b4004eb0 con 0x7f81d00af6c0 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81d0071bf0 con 0x7f81d00af6c0 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f81c00015c0 con 0x7f81d00af6c0 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.217+0000 7f81c67fc640 10 monclient(hunting): dump: 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:39.213 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 155293299 0 0) 0x7f81d0071bf0 con 0x7f81d00af6c0 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 >> v1:172.21.15.64:6790/0 conn(0x7f81d00ad6c0 legacy=0x7f81d014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 >> v1:172.21.15.64:6789/0 conn(0x7f81d0153090 legacy=0x7f81d01efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 monclient: found mon.b 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00707c0 con 0x7f81d00af6c0 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:39.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 10 monclient: _renew_subs 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c00038a0 con 0x7f81d00af6c0 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d014ddc0 con 0x7f81d00af6c0 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f81c0003cf0 con 0x7f81d00af6c0 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81c67fc640 10 monclient: dump: 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:39.215 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 10 monclient: _renew_subs 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d0151ef0 con 0x7f81d00af6c0 2024-03-29T11:31:39.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.218+0000 7f81d7c0d640 1 librados: init done 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.219+0000 7f81c7fff640 4 set_mon_vals no callback set 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.220+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f81c0003e50 con 0x7f81d00af6c0 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81d7c0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f81c0084c60 con 0x7f81d00af6c0 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81c67fc640 10 client.4528.objecter ms_dispatch 0x7f81d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81c67fc640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81c67fc640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81c67fc640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81d7c0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81d7c0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.221+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d014e110 con 0x7f81d00af6c0 2024-03-29T11:31:39.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.223+0000 7f81d59a9640 10 monclient: get_auth_request con 0x7f81b406dc90 auth_method 0 2024-03-29T11:31:39.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.223+0000 7f81c67fc640 10 client.4528.objecter ms_handle_connect 0x7f81b406dc90 2024-03-29T11:31:39.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.229+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81c0050540 con 0x7f81d00af6c0 2024-03-29T11:31:39.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.229+0000 7f81c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:39.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.229+0000 7f81c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:39.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.562+0000 7f81d7c0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.562+0000 7f81d7c0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.562+0000 7f81d7c0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:39.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.562+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81d0075f10 con 0x7f81d00af6c0 2024-03-29T11:31:39.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.563+0000 7f81c67fc640 1 -- 172.21.15.64:0/1666931235 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f81c0052470 con 0x7f81d00af6c0 2024-03-29T11:31:39.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.563+0000 7f81c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.563+0000 7f81c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:39.562 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:39.563 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:39.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.567+0000 7f81d7c0d640 10 librados: watch_flush enter 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.567+0000 7f81d7c0d640 10 librados: watch_flush exit 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.567+0000 7f81d7c0d640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.567+0000 7f81d7c0d640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.568+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 >> v1:172.21.15.136:6816/2312274743 conn(0x7f81b406dc90 legacy=0x7f81b4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.568+0000 7f81d7c0d640 10 monclient: shutdown 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.568+0000 7f81d7c0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.568+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 >> v1:172.21.15.136:6789/0 conn(0x7f81d00af6c0 legacy=0x7f81d01e41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.568+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 shutdown_connections 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.568+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 >> 172.21.15.64:0/1666931235 conn(0x7f81d00b35e0 msgr2=0x7f81d00abdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.569+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 shutdown_connections 2024-03-29T11:31:39.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.570+0000 7f81d7c0d640 1 -- 172.21.15.64:0/1666931235 wait complete. 2024-03-29T11:31:39.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:39.570+0000 7f81d7c0d640 1 librados: shutdown 2024-03-29T11:31:39.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:46.388 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:46.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.523+0000 7f31c880d640 10 monclient: get_monmap_and_config 2024-03-29T11:31:46.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.523+0000 7f31c880d640 10 monclient: build_initial_monmap 2024-03-29T11:31:46.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.523+0000 7f31c880d640 10 monclient: monmap: 2024-03-29T11:31:46.517 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:46.517 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:46.524065+0000 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:46.524065+0000 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:46.518 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.524+0000 7f31c880d640 1 Processor -- start 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.524+0000 7f31c880d640 1 -- start start 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.524+0000 7f31c880d640 10 monclient: init 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient(hunting): picked mon.noname-b con 0x7f31c41422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient(hunting): picked mon.noname-c con 0x7f31c4142690 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient(hunting): picked mon.noname-a con 0x7f31c40b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c4142c80 con 0x7f31c40b17d0 2024-03-29T11:31:46.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c40aae40 con 0x7f31c4142690 2024-03-29T11:31:46.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c413fbd0 con 0x7f31c41422b0 2024-03-29T11:31:46.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:46.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c880d640 10 monclient(hunting): authenticate will time out at 1642.795160s 2024-03-29T11:31:46.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c2d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f31c40b17d0 0x7f31c40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57868/0 (socket says 172.21.15.64:57868) 2024-03-29T11:31:46.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.525+0000 7f31c2d9d640 1 -- 172.21.15.64:0/299067242 learned_addr learned my addr 172.21.15.64:0/299067242 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:46.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.526+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058470959 0 0) 0x7f31c4142c80 con 0x7f31c40b17d0 2024-03-29T11:31:46.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.526+0000 7f31c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.526+0000 7f31c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.526+0000 7f31c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.526+0000 7f31c159a640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.526+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ac003520 con 0x7f31c40b17d0 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720286851 0 0) 0x7f31c40aae40 con 0x7f31c4142690 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c4142c80 con 0x7f31c4142690 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634042631 0 0) 0x7f31ac003520 con 0x7f31c40b17d0 2024-03-29T11:31:46.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.527+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 >> v1:172.21.15.64:6790/0 conn(0x7f31c41422b0 legacy=0x7f31c413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 >> v1:172.21.15.136:6789/0 conn(0x7f31c4142690 legacy=0x7f31c40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: _finish_hunting 0 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 1 monclient: found mon.noname-a 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c4151940 con 0x7f31c40b17d0 2024-03-29T11:31:46.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: _finish_auth 0 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: _check_auth_tickets 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f31b8001690 con 0x7f31c40b17d0 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: dump: 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.529 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c880d640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c880d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b80017f0 con 0x7f31c40b17d0 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 1 -- 172.21.15.64:0/299067242 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f31b8001bc0 con 0x7f31c40b17d0 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:46.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c159a640 10 monclient: dump: 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c880d640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.529+0000 7f31c880d640 4 set_mon_vals no callback set 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 10 monclient: shutdown 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:46.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 1 -- 172.21.15.64:0/299067242 >> v1:172.21.15.64:6789/0 conn(0x7f31c40b17d0 legacy=0x7f31c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 1 -- 172.21.15.64:0/299067242 shutdown_connections 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 1 -- 172.21.15.64:0/299067242 >> 172.21.15.64:0/299067242 conn(0x7f31c4066a00 msgr2=0x7f31c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 1 -- 172.21.15.64:0/299067242 shutdown_connections 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.531+0000 7f31c880d640 1 -- 172.21.15.64:0/299067242 wait complete. 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 10 monclient: build_initial_monmap 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 10 monclient: monmap: 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:46.532 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 1 librados: starting msgr at 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 1 librados: starting objecter 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 1 Processor -- start 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 1 -- start start 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 1 librados: setting wanted keys 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 1 librados: calling monclient init 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.532+0000 7f31c880d640 10 monclient: init 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient(hunting): picked mon.noname-a con 0x7f31c40b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient(hunting): picked mon.noname-b con 0x7f31c41422b0 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:46.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient(hunting): picked mon.noname-c con 0x7f31c4142690 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c41dac30 con 0x7f31c40b17d0 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c40acd60 con 0x7f31c41422b0 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c4132a80 con 0x7f31c4142690 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.533+0000 7f31c880d640 10 monclient(hunting): authenticate will time out at 1642.803706s 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31c259c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f31c40b17d0 0x7f31c40aba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33266/0 (socket says 172.21.15.64:33266) 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31c259c640 1 -- 172.21.15.64:0/2134136298 learned_addr learned my addr 172.21.15.64:0/2134136298 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:46.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 client.?.objecter ms_handle_connect 0x7f31c41422b0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183483052 0 0) 0x7f31c40acd60 con 0x7f31c41422b0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a4004eb0 con 0x7f31c41422b0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 client.?.objecter ms_handle_connect 0x7f31c40b17d0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.534+0000 7f31b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3893153746 0 0) 0x7f31a4004eb0 con 0x7f31c41422b0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31c40acd60 con 0x7f31c41422b0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f31b4001bc0 con 0x7f31c41422b0 2024-03-29T11:31:46.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:46.543 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392028695 0 0) 0x7f31c41dac30 con 0x7f31c40b17d0 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a4004eb0 con 0x7f31c40b17d0 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1746978703 0 0) 0x7f31c40acd60 con 0x7f31c41422b0 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.535+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 >> v1:172.21.15.64:6790/0 conn(0x7f31c4142690 legacy=0x7f31c412a090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 >> v1:172.21.15.64:6789/0 conn(0x7f31c40b17d0 legacy=0x7f31c40aba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 1 monclient: found mon.b 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c41db7d0 con 0x7f31c41422b0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31c880d640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31c880d640 10 monclient: _renew_subs 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31c880d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.536+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c412a7a0 con 0x7f31c41422b0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31c880d640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31c880d640 10 monclient: _renew_subs 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31c880d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:46.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c412fdf0 con 0x7f31c41422b0 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31c880d640 1 librados: init done 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b4003970 con 0x7f31c41422b0 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f31b4003c80 con 0x7f31c41422b0 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31b2ffd640 10 monclient: dump: 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:46.546 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.537+0000 7f31c0d99640 4 set_mon_vals no callback set 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.539+0000 7f31c880d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.539+0000 7f31c880d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.539+0000 7f31c880d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.539+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a8005080 con 0x7f31c41422b0 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.539+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f31b4019c50 con 0x7f31c41422b0 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.540+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f31b4083bf0 con 0x7f31c41422b0 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.540+0000 7f31b2ffd640 10 client.4534.objecter ms_dispatch 0x7f31c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.540+0000 7f31b2ffd640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.540+0000 7f31b2ffd640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:46.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.540+0000 7f31b2ffd640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:31:46.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.540+0000 7f31c259c640 10 monclient: get_auth_request con 0x7f31a406db80 auth_method 0 2024-03-29T11:31:46.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.541+0000 7f31b2ffd640 10 client.4534.objecter ms_handle_connect 0x7f31a406db80 2024-03-29T11:31:46.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.546+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31b404f4d0 con 0x7f31c41422b0 2024-03-29T11:31:46.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.546+0000 7f31b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.546+0000 7f31b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:46.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.855+0000 7f31c880d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:46.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.855+0000 7f31c880d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:46.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.855+0000 7f31c880d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:31:46.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.855+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31a8005330 con 0x7f31c41422b0 2024-03-29T11:31:46.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.856+0000 7f31b2ffd640 1 -- 172.21.15.64:0/2134136298 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f31b4051400 con 0x7f31c41422b0 2024-03-29T11:31:46.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.856+0000 7f31b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:46.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.856+0000 7f31b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:46.851 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:46.851 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.858+0000 7f31c880d640 10 librados: watch_flush enter 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.858+0000 7f31c880d640 10 librados: watch_flush exit 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.858+0000 7f31c880d640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.858+0000 7f31c880d640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.858+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 >> v1:172.21.15.136:6816/2312274743 conn(0x7f31a406db80 legacy=0x7f31a4070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.859+0000 7f31c880d640 10 monclient: shutdown 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.859+0000 7f31c880d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:46.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.859+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 >> v1:172.21.15.136:6789/0 conn(0x7f31c41422b0 legacy=0x7f31c41db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.859+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 shutdown_connections 2024-03-29T11:31:46.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.859+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 >> 172.21.15.64:0/2134136298 conn(0x7f31c4066a00 msgr2=0x7f31c40b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.860+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 shutdown_connections 2024-03-29T11:31:46.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.863+0000 7f31c880d640 1 -- 172.21.15.64:0/2134136298 wait complete. 2024-03-29T11:31:46.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:46.863+0000 7f31c880d640 1 librados: shutdown 2024-03-29T11:31:46.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:53.680 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:53.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.821+0000 7fd7a8f64640 10 monclient: get_monmap_and_config 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.821+0000 7fd7a8f64640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.821+0000 7fd7a8f64640 10 monclient: monmap: 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:31:53.822431+0000 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:31:53.822431+0000 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:53.816 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:53.817 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:31:53.817 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:31:53.817 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:53.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.822+0000 7fd7a8f64640 1 Processor -- start 2024-03-29T11:31:53.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.822+0000 7fd7a8f64640 1 -- start start 2024-03-29T11:31:53.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.822+0000 7fd7a8f64640 10 monclient: init 2024-03-29T11:31:53.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.824+0000 7fd7a8f64640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.824+0000 7fd7a8f64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:53.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.824+0000 7fd7a8f64640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a41422b0 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:53.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.824+0000 7fd7a8f64640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a4142690 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.824+0000 7fd7a8f64640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a40b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a8f64640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a4142c80 con 0x7fd7a40b17d0 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a1d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd7a4142690 0x7fd7a40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53160/0 (socket says 172.21.15.64:53160) 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a1d9b640 1 -- 172.21.15.64:0/3710839940 learned_addr learned my addr 172.21.15.64:0/3710839940 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a259c640 1 --1- 172.21.15.64:0/3710839940 >> v1:172.21.15.136:6789/0 conn(0x7fd7a41422b0 0x7fd7a413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:39806/0 (socket says 172.21.15.64:39806) 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40aae40 con 0x7fd7a41422b0 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a413fbd0 con 0x7fd7a4142690 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a8f64640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a8f64640 10 monclient(hunting): authenticate will time out at 1650.095257s 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437601111 0 0) 0x7fd7a413fbd0 con 0x7fd7a4142690 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c003520 con 0x7fd7a4142690 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61945134 0 0) 0x7fd7a40aae40 con 0x7fd7a41422b0 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a413fbd0 con 0x7fd7a41422b0 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3593854173 0 0) 0x7fd78c003520 con 0x7fd7a4142690 2024-03-29T11:31:53.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 >> v1:172.21.15.136:6789/0 conn(0x7fd7a41422b0 legacy=0x7fd7a413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.825+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 >> v1:172.21.15.64:6789/0 conn(0x7fd7a40b17d0 legacy=0x7fd7a40b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 1 monclient: found mon.noname-b 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a414f920 con 0x7fd7a4142690 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd798001690 con 0x7fd7a4142690 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a8f64640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a8f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:53.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a159a640 10 monclient: dump: 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.828 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.828+0000 7fd7a8f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd798001870 con 0x7fd7a4142690 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a159a640 1 -- 172.21.15.64:0/3710839940 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd798001c00 con 0x7fd7a4142690 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a159a640 10 monclient: dump: 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:53.830 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a8f64640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.829+0000 7fd7a8f64640 4 set_mon_vals no callback set 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 10 monclient: shutdown 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 >> v1:172.21.15.64:6790/0 conn(0x7fd7a4142690 legacy=0x7fd7a40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 shutdown_connections 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 >> 172.21.15.64:0/3710839940 conn(0x7fd7a4066a00 msgr2=0x7fd7a4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 shutdown_connections 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.830+0000 7fd7a8f64640 1 -- 172.21.15.64:0/3710839940 wait complete. 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.831+0000 7fd7a8f64640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.831+0000 7fd7a8f64640 10 monclient: monmap: 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:31:53.831 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.831+0000 7fd7a8f64640 1 librados: starting msgr at 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.831+0000 7fd7a8f64640 1 librados: starting objecter 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.832+0000 7fd7a8f64640 1 Processor -- start 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.832+0000 7fd7a8f64640 1 -- start start 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.832+0000 7fd7a8f64640 1 librados: setting wanted keys 2024-03-29T11:31:53.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.832+0000 7fd7a8f64640 1 librados: calling monclient init 2024-03-29T11:31:53.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.832+0000 7fd7a8f64640 10 monclient: init 2024-03-29T11:31:53.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.832+0000 7fd7a8f64640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.833+0000 7fd7a8f64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.833+0000 7fd7a8f64640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a40b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.833+0000 7fd7a259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd7a40b17d0 0x7fd7a4131260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54080/0 (socket says 172.21.15.64:54080) 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a41422b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a4142690 addr v1:172.21.15.136:6789/0 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40ac2f0 con 0x7fd7a41422b0 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a4130dd0 con 0x7fd7a4142690 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a4130aa0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a8f64640 10 monclient(hunting): authenticate will time out at 1650.104500s 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.834+0000 7fd7a259c640 1 -- 172.21.15.64:0/833680253 learned_addr learned my addr 172.21.15.64:0/833680253 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 client.?.objecter ms_handle_connect 0x7fd7a40b17d0 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562645253 0 0) 0x7fd7a4130aa0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784004eb0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286012053 0 0) 0x7fd784004eb0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a4130aa0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd794001a30 con 0x7fd7a40b17d0 2024-03-29T11:31:53.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:53.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3036078974 0 0) 0x7fd7a4130aa0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 >> v1:172.21.15.136:6789/0 conn(0x7fd7a4142690 legacy=0x7fd7a41e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 >> v1:172.21.15.64:6789/0 conn(0x7fd7a41422b0 legacy=0x7fd7a412dba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 1 monclient: found mon.c 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.835+0000 7fd792ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40ac540 con 0x7fd7a40b17d0 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 10 monclient: _renew_subs 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:53.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a4133cc0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 10 monclient: _renew_subs 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a412e2b0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a8f64640 1 librados: init done 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd794001d10 con 0x7fd7a40b17d0 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fd794004c30 con 0x7fd7a40b17d0 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd792ffd640 10 monclient: dump: 2024-03-29T11:31:53.841 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:31:53.842 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:31:53.842 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.842 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.836+0000 7fd7a0d99640 4 set_mon_vals no callback set 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.837+0000 7fd7a8f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.838+0000 7fd7a8f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.838+0000 7fd7a8f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.838+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd788005080 con 0x7fd7a40b17d0 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.838+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fd79401ac00 con 0x7fd7a40b17d0 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fd794084be0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd792ffd640 10 client.4532.objecter ms_dispatch 0x7fd7a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:53.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd792ffd640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:53.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd792ffd640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:53.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd792ffd640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:31:53.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd7a1d9b640 10 monclient: get_auth_request con 0x7fd78406db20 auth_method 0 2024-03-29T11:31:53.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.839+0000 7fd792ffd640 10 client.4532.objecter ms_handle_connect 0x7fd78406db20 2024-03-29T11:31:53.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.846+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7940502f0 con 0x7fd7a40b17d0 2024-03-29T11:31:53.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.846+0000 7fd792ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:53.846+0000 7fd792ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:54.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.164+0000 7fd7a8f64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:54.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.164+0000 7fd7a8f64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:54.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.164+0000 7fd7a8f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:31:54.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.164+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd788005330 con 0x7fd7a40b17d0 2024-03-29T11:31:54.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.165+0000 7fd792ffd640 1 -- 172.21.15.64:0/833680253 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fd7940523f0 con 0x7fd7a40b17d0 2024-03-29T11:31:54.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.165+0000 7fd792ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:54.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.165+0000 7fd792ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:54.160 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:31:54.160 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 10 librados: watch_flush enter 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 10 librados: watch_flush exit 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 >> v1:172.21.15.136:6816/2312274743 conn(0x7fd78406db20 legacy=0x7fd78406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 10 monclient: shutdown 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 >> v1:172.21.15.64:6790/0 conn(0x7fd7a40b17d0 legacy=0x7fd7a4131260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 shutdown_connections 2024-03-29T11:31:54.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 >> 172.21.15.64:0/833680253 conn(0x7fd7a4066a00 msgr2=0x7fd7a413d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 shutdown_connections 2024-03-29T11:31:54.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 -- 172.21.15.64:0/833680253 wait complete. 2024-03-29T11:31:54.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:31:54.168+0000 7fd7a8f64640 1 librados: shutdown 2024-03-29T11:31:54.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:00.986 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.140+0000 7fa9d78ae640 10 monclient: get_monmap_and_config 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.140+0000 7fa9d78ae640 10 monclient: build_initial_monmap 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.140+0000 7fa9d78ae640 10 monclient: monmap: 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:01.141194+0000 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:01.141194+0000 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:01.135 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:01.136 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:01.136 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.141+0000 7fa9d78ae640 1 Processor -- start 2024-03-29T11:32:01.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.141+0000 7fa9d78ae640 1 -- start start 2024-03-29T11:32:01.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.141+0000 7fa9d78ae640 10 monclient: init 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d01457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d013f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d00a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d00aaef0 con 0x7fa9d013f650 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0145bd0 con 0x7fa9d00a9d70 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0145eb0 con 0x7fa9d01457f0 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d5e4b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa9d00a9d70 0x7fa9d014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:46102/0 (socket says 172.21.15.64:46102) 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d5e4b640 1 -- 172.21.15.64:0/1625220445 learned_addr learned my addr 172.21.15.64:0/1625220445 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:01.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.142+0000 7fa9d78ae640 10 monclient(hunting): authenticate will time out at 1657.412686s 2024-03-29T11:32:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.146+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635376472 0 0) 0x7fa9d0145eb0 con 0x7fa9d01457f0 2024-03-29T11:32:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.146+0000 7fa9c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.146+0000 7fa9c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.146+0000 7fa9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.146+0000 7fa9c7fff640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:32:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.147+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc003520 con 0x7fa9d01457f0 2024-03-29T11:32:01.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.147+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3293414487 0 0) 0x7fa9bc003520 con 0x7fa9d01457f0 2024-03-29T11:32:01.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.147+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 >> v1:172.21.15.136:6789/0 conn(0x7fa9d00a9d70 legacy=0x7fa9d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.147+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 >> v1:172.21.15.64:6789/0 conn(0x7fa9d013f650 legacy=0x7fa9d00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 1 monclient: found mon.noname-b 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0146340 con 0x7fa9d01457f0 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:01.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa9cc001690 con 0x7fa9d01457f0 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.148+0000 7fa9c7fff640 10 monclient: dump: 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:01.146 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.149+0000 7fa9d78ae640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.149+0000 7fa9d78ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.149+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9cc001920 con 0x7fa9d01457f0 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.149+0000 7fa9c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.149+0000 7fa9d78ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.149+0000 7fa9d78ae640 4 set_mon_vals no callback set 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9c7fff640 1 -- 172.21.15.64:0/1625220445 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa9cc001d30 con 0x7fa9d01457f0 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9c7fff640 10 monclient: dump: 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.147 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9d78ae640 10 monclient: shutdown 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9d78ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.150+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1625220445 >> v1:172.21.15.64:6790/0 conn(0x7fa9d01457f0 legacy=0x7fa9d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.151+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1625220445 shutdown_connections 2024-03-29T11:32:01.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.151+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1625220445 >> 172.21.15.64:0/1625220445 conn(0x7fa9d0066a00 msgr2=0x7fa9d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.151+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1625220445 shutdown_connections 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.151+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1625220445 wait complete. 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.152+0000 7fa9d78ae640 10 monclient: build_initial_monmap 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.152+0000 7fa9d78ae640 10 monclient: monmap: 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:01.150 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 1 librados: starting msgr at 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 1 librados: starting objecter 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 1 Processor -- start 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 1 -- start start 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 1 librados: setting wanted keys 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 1 librados: calling monclient init 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.153+0000 7fa9d78ae640 10 monclient: init 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.154+0000 7fa9d78ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:01.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.156+0000 7fa9d78ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.157+0000 7fa9d78ae640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d00a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.157+0000 7fa9d78ae640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d013f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.157+0000 7fa9d78ae640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d01457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.157+0000 7fa9d78ae640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0142140 con 0x7fa9d01457f0 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.157+0000 7fa9d78ae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0142400 con 0x7fa9d013f650 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.158+0000 7fa9d78ae640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0141a40 con 0x7fa9d00a9d70 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.158+0000 7fa9d78ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:01.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.158+0000 7fa9d78ae640 10 monclient(hunting): authenticate will time out at 1657.427902s 2024-03-29T11:32:01.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.159+0000 7fa9d4e49640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa9d013f650 0x7fa9d01db520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:47520/0 (socket says 172.21.15.64:47520) 2024-03-29T11:32:01.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.159+0000 7fa9d4e49640 1 -- 172.21.15.64:0/1488083820 learned_addr learned my addr 172.21.15.64:0/1488083820 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:01.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.159+0000 7fa9c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa9d013f650 2024-03-29T11:32:01.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.159+0000 7fa9c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:01.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa9d00a9d70 2024-03-29T11:32:01.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:01.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514925362 0 0) 0x7fa9d0142400 con 0x7fa9d013f650 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b4004eb0 con 0x7fa9d013f650 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948936874 0 0) 0x7fa9d0141a40 con 0x7fa9d00a9d70 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.160+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0142400 con 0x7fa9d00a9d70 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175328198 0 0) 0x7fa9b4004eb0 con 0x7fa9d013f650 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d0141a40 con 0x7fa9d013f650 2024-03-29T11:32:01.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa9c0001960 con 0x7fa9d013f650 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3788583918 0 0) 0x7fa9d0142400 con 0x7fa9d00a9d70 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b4004eb0 con 0x7fa9d00a9d70 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa9cc001d30 con 0x7fa9d00a9d70 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:01.159 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 305911521 0 0) 0x7fa9d0141a40 con 0x7fa9d013f650 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 >> v1:172.21.15.64:6790/0 conn(0x7fa9d00a9d70 legacy=0x7fa9d0142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 >> v1:172.21.15.64:6789/0 conn(0x7fa9d01457f0 legacy=0x7fa9d013b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 monclient: found mon.b 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0140280 con 0x7fa9d013f650 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.161+0000 7fa9c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:32:01.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 10 monclient: _renew_subs 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d0140530 con 0x7fa9d013f650 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c00035f0 con 0x7fa9d013f650 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa9c0003940 con 0x7fa9d013f650 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c5ffb640 10 monclient: dump: 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:01.161 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 10 monclient: _renew_subs 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d01350e0 con 0x7fa9d013f650 2024-03-29T11:32:01.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9d78ae640 1 librados: init done 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.162+0000 7fa9c77fe640 4 set_mon_vals no callback set 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.163+0000 7fa9d78ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.164+0000 7fa9d78ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.164+0000 7fa9d78ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.164+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0005080 con 0x7fa9d013f650 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.165+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fa9c0003aa0 con 0x7fa9d013f650 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.165+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fa9c0084a40 con 0x7fa9d013f650 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.165+0000 7fa9c5ffb640 10 client.4543.objecter ms_dispatch 0x7fa9d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.165+0000 7fa9c5ffb640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.165+0000 7fa9c5ffb640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.165+0000 7fa9c5ffb640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.166+0000 7fa9d564a640 10 monclient: get_auth_request con 0x7fa9b406e050 auth_method 0 2024-03-29T11:32:01.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.167+0000 7fa9c5ffb640 10 client.4543.objecter ms_handle_connect 0x7fa9b406e050 2024-03-29T11:32:01.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.176+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9c0050320 con 0x7fa9d013f650 2024-03-29T11:32:01.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.176+0000 7fa9c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:01.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.176+0000 7fa9c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:01.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.500+0000 7fa9d78ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:01.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.500+0000 7fa9d78ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:01.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.500+0000 7fa9d78ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:01.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.500+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9b0005330 con 0x7fa9d013f650 2024-03-29T11:32:01.497 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:01.497 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:01.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.502+0000 7fa9c5ffb640 1 -- 172.21.15.64:0/1488083820 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fa9c0052250 con 0x7fa9d013f650 2024-03-29T11:32:01.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.502+0000 7fa9c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:01.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.502+0000 7fa9c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:01.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 10 librados: watch_flush enter 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 10 librados: watch_flush exit 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 10 client.4543.objecter successfully canceled tick 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 >> v1:172.21.15.136:6816/2312274743 conn(0x7fa9b406e050 legacy=0x7fa9b4070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 10 monclient: shutdown 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 >> v1:172.21.15.136:6789/0 conn(0x7fa9d013f650 legacy=0x7fa9d01db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 shutdown_connections 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.506+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 >> 172.21.15.64:0/1488083820 conn(0x7fa9d0066a00 msgr2=0x7fa9d00b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.507+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 shutdown_connections 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.507+0000 7fa9d78ae640 1 -- 172.21.15.64:0/1488083820 wait complete. 2024-03-29T11:32:01.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:01.507+0000 7fa9d78ae640 1 librados: shutdown 2024-03-29T11:32:01.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:08.325 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:08.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.490+0000 7f04449c8640 10 monclient: get_monmap_and_config 2024-03-29T11:32:08.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.490+0000 7f04449c8640 10 monclient: build_initial_monmap 2024-03-29T11:32:08.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.490+0000 7f04449c8640 10 monclient: monmap: 2024-03-29T11:32:08.487 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:08.492501+0000 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:08.492501+0000 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:08.488 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.491+0000 7f04449c8640 1 Processor -- start 2024-03-29T11:32:08.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.491+0000 7f04449c8640 1 -- start start 2024-03-29T11:32:08.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.491+0000 7f04449c8640 10 monclient: init 2024-03-29T11:32:08.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:08.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient(hunting): picked mon.noname-c con 0x7f04400b4100 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient(hunting): picked mon.noname-b con 0x7f044013ee90 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient(hunting): picked mon.noname-a con 0x7f04400af900 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04400afd10 con 0x7f04400af900 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04400af6f0 con 0x7f04400b4100 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04400aac80 con 0x7f044013ee90 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.492+0000 7f04449c8640 10 monclient(hunting): authenticate will time out at 1664.763800s 2024-03-29T11:32:08.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.495+0000 7f043dd9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f044013ee90 0x7f044013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40773/0 (socket says 172.21.15.64:40773) 2024-03-29T11:32:08.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.495+0000 7f043dd9b640 1 -- 172.21.15.64:0/2514151572 learned_addr learned my addr 172.21.15.64:0/2514151572 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480648719 0 0) 0x7f04400aac80 con 0x7f044013ee90 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0428003520 con 0x7f044013ee90 2024-03-29T11:32:08.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348069489 0 0) 0x7f0428003520 con 0x7f044013ee90 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 >> v1:172.21.15.136:6789/0 conn(0x7f04400b4100 legacy=0x7f04400adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 >> v1:172.21.15.64:6789/0 conn(0x7f04400af900 legacy=0x7f044014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 monclient: found mon.noname-b 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.499+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04400aaf60 con 0x7f044013ee90 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 10 monclient: _finish_auth 0 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:32:08.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f04449c8640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f04449c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0434001690 con 0x7f044013ee90 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 10 monclient: dump: 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:08.500 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f04449c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 1 -- 172.21.15.64:0/2514151572 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0434001920 con 0x7f044013ee90 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f043d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f04449c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.500+0000 7f04449c8640 4 set_mon_vals no callback set 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.501+0000 7f04449c8640 10 monclient: shutdown 2024-03-29T11:32:08.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.501+0000 7f04449c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:08.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.501+0000 7f04449c8640 1 -- 172.21.15.64:0/2514151572 >> v1:172.21.15.64:6790/0 conn(0x7f044013ee90 legacy=0x7f044013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.501+0000 7f04449c8640 1 -- 172.21.15.64:0/2514151572 shutdown_connections 2024-03-29T11:32:08.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.501+0000 7f04449c8640 1 -- 172.21.15.64:0/2514151572 >> 172.21.15.64:0/2514151572 conn(0x7f04400b35e0 msgr2=0x7f04400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.502+0000 7f04449c8640 1 -- 172.21.15.64:0/2514151572 shutdown_connections 2024-03-29T11:32:08.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.502+0000 7f04449c8640 1 -- 172.21.15.64:0/2514151572 wait complete. 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 10 monclient: build_initial_monmap 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 10 monclient: monmap: 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:08.503 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 1 librados: starting msgr at 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 1 librados: starting objecter 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 1 Processor -- start 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 1 -- start start 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 1 librados: setting wanted keys 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 1 librados: calling monclient init 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.503+0000 7f04449c8640 10 monclient: init 2024-03-29T11:32:08.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient(hunting): picked mon.noname-c con 0x7f04400af900 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient(hunting): picked mon.noname-a con 0x7f04400b4100 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient(hunting): picked mon.noname-b con 0x7f044006eeb0 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044013b100 con 0x7f04400b4100 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044013ac70 con 0x7f044006eeb0 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04401442f0 con 0x7f04400af900 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:08.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.504+0000 7f04449c8640 10 monclient(hunting): authenticate will time out at 1664.775547s 2024-03-29T11:32:08.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.510+0000 7f043dd9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f04400b4100 0x7f044014be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:44063/0 (socket says 172.21.15.64:44063) 2024-03-29T11:32:08.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f043dd9b640 1 -- 172.21.15.64:0/2550704977 learned_addr learned my addr 172.21.15.64:0/2550704977 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:08.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 client.?.objecter ms_handle_connect 0x7f04400b4100 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 client.?.objecter ms_handle_connect 0x7f044006eeb0 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547019322 0 0) 0x7f04400aaf60 con 0x7f04400b4100 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0420004eb0 con 0x7f04400b4100 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254159916 0 0) 0x7f044013ac70 con 0x7f044006eeb0 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.511+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04400aaf60 con 0x7f044006eeb0 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506163428 0 0) 0x7f0420004eb0 con 0x7f04400b4100 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f044013ac70 con 0x7f04400b4100 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2434496732 0 0) 0x7f04400aaf60 con 0x7f044006eeb0 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0420004eb0 con 0x7f044006eeb0 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0434001930 con 0x7f04400b4100 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): dump: 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.513 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0438001730 con 0x7f044006eeb0 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.512+0000 7f042effd640 10 monclient(hunting): dump: 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.514 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3695344734 0 0) 0x7f0420004eb0 con 0x7f044006eeb0 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 >> v1:172.21.15.64:6790/0 conn(0x7f04400af900 legacy=0x7f044006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 >> v1:172.21.15.64:6789/0 conn(0x7f04400b4100 legacy=0x7f044014be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 1 monclient: found mon.b 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:08.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04401445d0 con 0x7f044006eeb0 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 10 monclient: _finish_auth 0 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0438001c40 con 0x7f044006eeb0 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.513+0000 7f042ffff640 4 set_mon_vals no callback set 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.514+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0438003790 con 0x7f044006eeb0 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.514+0000 7f042effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.514+0000 7f042effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.514+0000 7f042effd640 10 monclient: dump: 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:08.516 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.517 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:08.517 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:08.517 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:08.517 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:08.517 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 10 monclient: _renew_subs 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04401448b0 con 0x7f044006eeb0 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 10 monclient: _renew_subs 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:08.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0440072590 con 0x7f044006eeb0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.516+0000 7f04449c8640 1 librados: init done 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.518+0000 7f04449c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.518+0000 7f04449c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.518+0000 7f04449c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.518+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044014cc60 con 0x7f044006eeb0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.519+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f0438019760 con 0x7f044006eeb0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f04380834b0 con 0x7f044006eeb0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f042effd640 10 client.4546.objecter ms_dispatch 0x7f0440000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f042effd640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f042effd640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f042effd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f043e59c640 10 monclient: get_auth_request con 0x7f042006deb0 auth_method 0 2024-03-29T11:32:08.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.520+0000 7f042effd640 10 client.4546.objecter ms_handle_connect 0x7f042006deb0 2024-03-29T11:32:08.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.527+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f043804ebc0 con 0x7f044006eeb0 2024-03-29T11:32:08.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.527+0000 7f042effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:08.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.527+0000 7f042effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:08.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.856+0000 7f04449c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:08.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.856+0000 7f04449c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:08.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.856+0000 7f04449c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:08.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.856+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0440072590 con 0x7f044006eeb0 2024-03-29T11:32:08.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.857+0000 7f042effd640 1 -- 172.21.15.64:0/2550704977 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f0440072590 con 0x7f044006eeb0 2024-03-29T11:32:08.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.857+0000 7f042effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:08.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.857+0000 7f042effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:08.859 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:08.859 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 10 librados: watch_flush enter 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 10 librados: watch_flush exit 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 >> v1:172.21.15.136:6816/2312274743 conn(0x7f042006deb0 legacy=0x7f0420070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 10 monclient: shutdown 2024-03-29T11:32:08.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:08.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 >> v1:172.21.15.136:6789/0 conn(0x7f044006eeb0 legacy=0x7f044014c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 shutdown_connections 2024-03-29T11:32:08.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.864+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 >> 172.21.15.64:0/2550704977 conn(0x7f04400b35e0 msgr2=0x7f04400ad270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.868+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 shutdown_connections 2024-03-29T11:32:08.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.869+0000 7f04449c8640 1 -- 172.21.15.64:0/2550704977 wait complete. 2024-03-29T11:32:08.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:08.869+0000 7f04449c8640 1 librados: shutdown 2024-03-29T11:32:08.883 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:15.688 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:15.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.843+0000 7f16c138e640 10 monclient: get_monmap_and_config 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.843+0000 7f16c138e640 10 monclient: build_initial_monmap 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.843+0000 7f16c138e640 10 monclient: monmap: 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:15.844846+0000 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:15.844846+0000 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:15.839 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:15.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.844+0000 7f16c138e640 1 Processor -- start 2024-03-29T11:32:15.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.844+0000 7f16c138e640 1 -- start start 2024-03-29T11:32:15.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.844+0000 7f16c138e640 10 monclient: init 2024-03-29T11:32:15.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient(hunting): picked mon.noname-c con 0x7f16b4006a90 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient(hunting): picked mon.noname-b con 0x7f16b4030520 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient(hunting): picked mon.noname-a con 0x7f16b40cbd70 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40cc200 con 0x7f16b40cbd70 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40356a0 con 0x7f16b4006a90 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b4037250 con 0x7f16b4030520 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:15.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.845+0000 7f16c138e640 10 monclient(hunting): authenticate will time out at 1672.115737s 2024-03-29T11:32:15.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.848+0000 7f16bb7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f16b40cbd70 0x7f16b40d7610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58685/0 (socket says 172.21.15.64:58685) 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.848+0000 7f16bb7fe640 1 -- 172.21.15.64:0/3097641859 learned_addr learned my addr 172.21.15.64:0/3097641859 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.849+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265279677 0 0) 0x7f16b40cc200 con 0x7f16b40cbd70 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.849+0000 7f16b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.849+0000 7f16b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.849+0000 7f16b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.849+0000 7f16b9ffb640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.849+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003520 con 0x7f16b40cbd70 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.850+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2775924610 0 0) 0x7f16a4003520 con 0x7f16b40cbd70 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.850+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 >> v1:172.21.15.64:6790/0 conn(0x7f16b4030520 legacy=0x7f16b4033430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 >> v1:172.21.15.136:6789/0 conn(0x7f16b4006a90 legacy=0x7f16b4035b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:15.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4033b40 con 0x7f16b40cbd70 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f16bc04f730 con 0x7f16b40cbd70 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: dump: 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:15.848 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16bc0008d0 con 0x7f16b40cbd70 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 1 -- 172.21.15.64:0/3097641859 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f16bc0758b0 con 0x7f16b40cbd70 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16b9ffb640 10 monclient: dump: 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.849 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:15.850 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:15.850 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:15.850 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:15.850 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:15.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16c138e640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16c138e640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.851+0000 7f16c138e640 4 set_mon_vals no callback set 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.852+0000 7f16c138e640 10 monclient: shutdown 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.852+0000 7f16c138e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.852+0000 7f16c138e640 1 -- 172.21.15.64:0/3097641859 >> v1:172.21.15.64:6789/0 conn(0x7f16b40cbd70 legacy=0x7f16b40d7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.853+0000 7f16c138e640 1 -- 172.21.15.64:0/3097641859 shutdown_connections 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.853+0000 7f16c138e640 1 -- 172.21.15.64:0/3097641859 >> 172.21.15.64:0/3097641859 conn(0x7f16b4005c50 msgr2=0x7f16b4006060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.853+0000 7f16c138e640 1 -- 172.21.15.64:0/3097641859 shutdown_connections 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.853+0000 7f16c138e640 1 -- 172.21.15.64:0/3097641859 wait complete. 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.854+0000 7f16c138e640 10 monclient: build_initial_monmap 2024-03-29T11:32:15.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.854+0000 7f16c138e640 10 monclient: monmap: 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.855+0000 7f16c138e640 1 librados: starting msgr at 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.855+0000 7f16c138e640 1 librados: starting objecter 2024-03-29T11:32:15.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.855+0000 7f16c138e640 1 Processor -- start 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.856+0000 7f16c138e640 1 -- start start 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.856+0000 7f16c138e640 1 librados: setting wanted keys 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.856+0000 7f16c138e640 1 librados: calling monclient init 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.856+0000 7f16c138e640 10 monclient: init 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.857+0000 7f16c138e640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.857+0000 7f16c138e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.857+0000 7f16c138e640 10 monclient(hunting): picked mon.noname-b con 0x7f16b4006a90 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 10 monclient(hunting): picked mon.noname-c con 0x7f16b40cbd70 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:15.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 10 monclient(hunting): picked mon.noname-a con 0x7f16b416ca30 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:15.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b402fb20 con 0x7f16b416ca30 2024-03-29T11:32:15.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b402fe00 con 0x7f16b4006a90 2024-03-29T11:32:15.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b402ee80 con 0x7f16b40cbd70 2024-03-29T11:32:15.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:15.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.858+0000 7f16c138e640 10 monclient(hunting): authenticate will time out at 1672.127936s 2024-03-29T11:32:15.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.861+0000 7f16ba7fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f16b40cbd70 0x7f16b4169b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:44247/0 (socket says 172.21.15.64:44247) 2024-03-29T11:32:15.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.861+0000 7f16ba7fc640 1 -- 172.21.15.64:0/2869128726 learned_addr learned my addr 172.21.15.64:0/2869128726 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.861+0000 7f16a37fe640 10 client.?.objecter ms_handle_connect 0x7f16b40cbd70 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.861+0000 7f16a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 10 client.?.objecter ms_handle_connect 0x7f16b4006a90 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901193829 0 0) 0x7f16b402ee80 con 0x7f16b40cbd70 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a8004eb0 con 0x7f16b40cbd70 2024-03-29T11:32:15.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888389845 0 0) 0x7f16a8004eb0 con 0x7f16b40cbd70 2024-03-29T11:32:15.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.862+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b402ee80 con 0x7f16b40cbd70 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f16b00018b0 con 0x7f16b40cbd70 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient(hunting): dump: 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:15.863 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 408170380 0 0) 0x7f16b402ee80 con 0x7f16b40cbd70 2024-03-29T11:32:15.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 >> v1:172.21.15.136:6789/0 conn(0x7f16b4006a90 legacy=0x7f16b402e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 >> v1:172.21.15.64:6789/0 conn(0x7f16b416ca30 legacy=0x7f16b4174d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 1 monclient: found mon.c 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b402f010 con 0x7f16b40cbd70 2024-03-29T11:32:15.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:15.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:15.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.863+0000 7f16a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:15.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b0001a80 con 0x7f16b40cbd70 2024-03-29T11:32:15.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:15.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f16b0001d10 con 0x7f16b40cbd70 2024-03-29T11:32:15.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16a37fe640 10 monclient: dump: 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:15.870 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.864+0000 7f16b8ff9640 4 set_mon_vals no callback set 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 10 monclient: _renew_subs 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b41556f0 con 0x7f16b40cbd70 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 10 monclient: _renew_subs 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:15.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b416a440 con 0x7f16b40cbd70 2024-03-29T11:32:15.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.865+0000 7f16c138e640 1 librados: init done 2024-03-29T11:32:15.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.867+0000 7f16c138e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:15.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.867+0000 7f16c138e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:15.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.867+0000 7f16c138e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:15.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.867+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b4155a90 con 0x7f16b40cbd70 2024-03-29T11:32:15.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.869+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f16b00038a0 con 0x7f16b40cbd70 2024-03-29T11:32:15.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.870+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f16b0084880 con 0x7f16b40cbd70 2024-03-29T11:32:15.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.870+0000 7f16a37fe640 10 client.4544.objecter ms_dispatch 0x7f16bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:15.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.870+0000 7f16a37fe640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:15.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.870+0000 7f16a37fe640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:15.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.870+0000 7f16a37fe640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:32:15.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.873+0000 7f16baffd640 10 monclient: get_auth_request con 0x7f16a806dd50 auth_method 0 2024-03-29T11:32:15.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.873+0000 7f16a37fe640 10 client.4544.objecter ms_handle_connect 0x7f16a806dd50 2024-03-29T11:32:15.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.878+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16b004ff90 con 0x7f16b40cbd70 2024-03-29T11:32:15.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.878+0000 7f16a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:15.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:15.878+0000 7f16a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:16.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.207+0000 7f16c138e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:16.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.207+0000 7f16c138e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:16.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.207+0000 7f16c138e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:16.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.207+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16b4155880 con 0x7f16b40cbd70 2024-03-29T11:32:16.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.209+0000 7f16a37fe640 1 -- 172.21.15.64:0/2869128726 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f16b0052090 con 0x7f16b40cbd70 2024-03-29T11:32:16.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.209+0000 7f16a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:16.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.209+0000 7f16a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:16.205 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:16.205 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:16.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.214+0000 7f16c138e640 10 librados: watch_flush enter 2024-03-29T11:32:16.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.214+0000 7f16c138e640 10 librados: watch_flush exit 2024-03-29T11:32:16.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.214+0000 7f16c138e640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:32:16.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.214+0000 7f16c138e640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:32:16.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.215+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 >> v1:172.21.15.136:6816/2312274743 conn(0x7f16a806dd50 legacy=0x7f16a8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.215+0000 7f16c138e640 10 monclient: shutdown 2024-03-29T11:32:16.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.216+0000 7f16c138e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:16.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.216+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 >> v1:172.21.15.64:6790/0 conn(0x7f16b40cbd70 legacy=0x7f16b4169b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:16.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.216+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 shutdown_connections 2024-03-29T11:32:16.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.217+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 >> 172.21.15.64:0/2869128726 conn(0x7f16b4005c50 msgr2=0x7f16b4004400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:16.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.217+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 shutdown_connections 2024-03-29T11:32:16.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.217+0000 7f16c138e640 1 -- 172.21.15.64:0/2869128726 wait complete. 2024-03-29T11:32:16.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:16.217+0000 7f16c138e640 1 librados: shutdown 2024-03-29T11:32:16.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:23.038 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:23.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.195+0000 7fc58ddcf640 10 monclient: get_monmap_and_config 2024-03-29T11:32:23.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.195+0000 7fc58ddcf640 10 monclient: build_initial_monmap 2024-03-29T11:32:23.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.195+0000 7fc58ddcf640 10 monclient: monmap: 2024-03-29T11:32:23.190 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:23.196111+0000 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:23.196111+0000 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:23.191 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:23.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.200+0000 7fc58ddcf640 1 Processor -- start 2024-03-29T11:32:23.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.200+0000 7fc58ddcf640 1 -- start start 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.200+0000 7fc58ddcf640 10 monclient: init 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc5880b4a30 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc5880b1220 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc588145a70 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b0bb0 con 0x7fc5880b1220 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b0e00 con 0x7fc5880b4a30 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880afd50 con 0x7fc588145a70 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:23.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.201+0000 7fc58ddcf640 10 monclient(hunting): authenticate will time out at 1679.471741s 2024-03-29T11:32:23.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.209+0000 7fc5877fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc5880b4a30 0x7fc5880b3f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:42475/0 (socket says 172.21.15.64:42475) 2024-03-29T11:32:23.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.209+0000 7fc5877fe640 1 -- 172.21.15.64:0/179282696 learned_addr learned my addr 172.21.15.64:0/179282696 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914146333 0 0) 0x7fc5880b0e00 con 0x7fc5880b4a30 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc570003520 con 0x7fc5880b4a30 2024-03-29T11:32:23.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294188514 0 0) 0x7fc570003520 con 0x7fc5880b4a30 2024-03-29T11:32:23.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 >> v1:172.21.15.64:6790/0 conn(0x7fc588145a70 legacy=0x7fc588151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:23.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.210+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 >> v1:172.21.15.64:6789/0 conn(0x7fc5880b1220 legacy=0x7fc5880af440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:23.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:23.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 1 monclient: found mon.noname-c 2024-03-29T11:32:23.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:23.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:32:23.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5880b1660 con 0x7fc5880b4a30 2024-03-29T11:32:23.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc57c001690 con 0x7fc5880b4a30 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: dump: 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:23.208 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:23.209 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:23.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc58ddcf640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:32:23.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc58ddcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:23.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 1 -- 172.21.15.64:0/179282696 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc57c001920 con 0x7fc5880b4a30 2024-03-29T11:32:23.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc5867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:23.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc58ddcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:23.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.212+0000 7fc58ddcf640 4 set_mon_vals no callback set 2024-03-29T11:32:23.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 10 monclient: shutdown 2024-03-29T11:32:23.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:23.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 1 -- 172.21.15.64:0/179282696 >> v1:172.21.15.136:6789/0 conn(0x7fc5880b4a30 legacy=0x7fc5880b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 1 -- 172.21.15.64:0/179282696 shutdown_connections 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 1 -- 172.21.15.64:0/179282696 >> 172.21.15.64:0/179282696 conn(0x7fc58806c8a0 msgr2=0x7fc58806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 1 -- 172.21.15.64:0/179282696 shutdown_connections 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.213+0000 7fc58ddcf640 1 -- 172.21.15.64:0/179282696 wait complete. 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.214+0000 7fc58ddcf640 10 monclient: build_initial_monmap 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.214+0000 7fc58ddcf640 10 monclient: monmap: 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:23.212 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.215+0000 7fc58ddcf640 1 librados: starting msgr at 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.215+0000 7fc58ddcf640 1 librados: starting objecter 2024-03-29T11:32:23.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.215+0000 7fc58ddcf640 1 Processor -- start 2024-03-29T11:32:23.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 1 -- start start 2024-03-29T11:32:23.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 1 librados: setting wanted keys 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 1 librados: calling monclient init 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient: init 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc5880b1220 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc5880b4a30 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc588145a70 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5881dbb40 con 0x7fc5880b1220 2024-03-29T11:32:23.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5881db7f0 con 0x7fc5880b4a30 2024-03-29T11:32:23.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880719d0 con 0x7fc588145a70 2024-03-29T11:32:23.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:23.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.216+0000 7fc58ddcf640 10 monclient(hunting): authenticate will time out at 1679.486831s 2024-03-29T11:32:23.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.222+0000 7fc586ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc5880b4a30 0x7fc58806e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:42559/0 (socket says 172.21.15.64:42559) 2024-03-29T11:32:23.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.222+0000 7fc586ffd640 1 -- 172.21.15.64:0/1019252214 learned_addr learned my addr 172.21.15.64:0/1019252214 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:23.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.222+0000 7fc56ffff640 10 client.?.objecter ms_handle_connect 0x7fc5880b4a30 2024-03-29T11:32:23.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.222+0000 7fc56ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:23.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.223+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488807472 0 0) 0x7fc5881db7f0 con 0x7fc5880b4a30 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.223+0000 7fc56ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.223+0000 7fc56ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.223+0000 7fc56ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.223+0000 7fc56ffff640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.223+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574004eb0 con 0x7fc5880b4a30 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1027793762 0 0) 0x7fc574004eb0 con 0x7fc5880b4a30 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc574005300 con 0x7fc5880b4a30 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc5800043c0 con 0x7fc5880b4a30 2024-03-29T11:32:23.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient(hunting): dump: 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:23.222 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2902613156 0 0) 0x7fc574005300 con 0x7fc5880b4a30 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 >> v1:172.21.15.64:6790/0 conn(0x7fc588145a70 legacy=0x7fc58806efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 >> v1:172.21.15.64:6789/0 conn(0x7fc5880b1220 legacy=0x7fc5881dc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 monclient: found mon.b 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:23.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588071b60 con 0x7fc5880b4a30 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient: _finish_auth 0 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc580001250 con 0x7fc5880b4a30 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.224+0000 7fc56ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.225+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc5800043c0 con 0x7fc5880b4a30 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.225+0000 7fc56ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.225+0000 7fc56ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.225+0000 7fc56ffff640 10 monclient: dump: 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:23.224 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:23.225 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc585ffb640 4 set_mon_vals no callback set 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 10 monclient: _renew_subs 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5881e7030 con 0x7fc5880b4a30 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:23.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 10 monclient: _renew_subs 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5881e73a0 con 0x7fc5880b4a30 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.226+0000 7fc58ddcf640 1 librados: init done 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.231+0000 7fc58ddcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.231+0000 7fc58ddcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.232+0000 7fc58ddcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:23.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.232+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc588075f10 con 0x7fc5880b4a30 2024-03-29T11:32:23.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.232+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fc580001580 con 0x7fc5880b4a30 2024-03-29T11:32:23.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.233+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fc580054740 con 0x7fc5880b4a30 2024-03-29T11:32:23.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.233+0000 7fc56ffff640 10 client.4555.objecter ms_dispatch 0x7fc588000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:23.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.233+0000 7fc56ffff640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:23.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.233+0000 7fc56ffff640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:23.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.233+0000 7fc56ffff640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-29T11:32:23.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.233+0000 7fc5877fe640 10 monclient: get_auth_request con 0x7fc57406e210 auth_method 0 2024-03-29T11:32:23.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.236+0000 7fc56ffff640 10 client.4555.objecter ms_handle_connect 0x7fc57406e210 2024-03-29T11:32:23.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.240+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5800509e0 con 0x7fc5880b4a30 2024-03-29T11:32:23.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.240+0000 7fc56ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:23.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.240+0000 7fc56ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:23.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.585+0000 7fc58ddcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:23.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.585+0000 7fc58ddcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:23.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.586+0000 7fc58ddcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:23.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.586+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc588071cf0 con 0x7fc5880b4a30 2024-03-29T11:32:23.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.587+0000 7fc56ffff640 1 -- 172.21.15.64:0/1019252214 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fc5800509e0 con 0x7fc5880b4a30 2024-03-29T11:32:23.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.587+0000 7fc56ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:23.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.587+0000 7fc56ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:23.584 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:23.584 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:23.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.592+0000 7fc58ddcf640 10 librados: watch_flush enter 2024-03-29T11:32:23.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.592+0000 7fc58ddcf640 10 librados: watch_flush exit 2024-03-29T11:32:23.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.592+0000 7fc58ddcf640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-29T11:32:23.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.592+0000 7fc58ddcf640 10 client.4555.objecter successfully canceled tick 2024-03-29T11:32:23.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.593+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 >> v1:172.21.15.136:6816/2312274743 conn(0x7fc57406e210 legacy=0x7fc5740706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.593+0000 7fc58ddcf640 10 monclient: shutdown 2024-03-29T11:32:23.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.593+0000 7fc58ddcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:23.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.593+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 >> v1:172.21.15.136:6789/0 conn(0x7fc5880b4a30 legacy=0x7fc58806e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.593+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 shutdown_connections 2024-03-29T11:32:23.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.593+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 >> 172.21.15.64:0/1019252214 conn(0x7fc58806c8a0 msgr2=0x7fc5880af130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:23.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.597+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 shutdown_connections 2024-03-29T11:32:23.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.598+0000 7fc58ddcf640 1 -- 172.21.15.64:0/1019252214 wait complete. 2024-03-29T11:32:23.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:23.598+0000 7fc58ddcf640 1 librados: shutdown 2024-03-29T11:32:23.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:30.430 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:30.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.597+0000 7f25979e8640 10 monclient: get_monmap_and_config 2024-03-29T11:32:30.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.597+0000 7f25979e8640 10 monclient: build_initial_monmap 2024-03-29T11:32:30.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.597+0000 7f25979e8640 10 monclient: monmap: 2024-03-29T11:32:30.592 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:30.592 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:30.598052+0000 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:30.598052+0000 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:30.593 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.599+0000 7f25979e8640 1 Processor -- start 2024-03-29T11:32:30.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.599+0000 7f25979e8640 1 -- start start 2024-03-29T11:32:30.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.599+0000 7f25979e8640 10 monclient: init 2024-03-29T11:32:30.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:30.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:30.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient(hunting): picked mon.noname-b con 0x7f25900b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:30.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient(hunting): picked mon.noname-a con 0x7f2590071eb0 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient(hunting): picked mon.noname-c con 0x7f2590143d10 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259006ef80 con 0x7f2590071eb0 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259006f260 con 0x7f2590143d10 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900b40c0 con 0x7f25900b1f60 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f25979e8640 10 monclient(hunting): authenticate will time out at 1686.870235s 2024-03-29T11:32:30.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f2595784640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f25900b1f60 0x7f2590143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53872/0 (socket says 172.21.15.64:53872) 2024-03-29T11:32:30.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.600+0000 7f2595784640 1 -- 172.21.15.64:0/2780022514 learned_addr learned my addr 172.21.15.64:0/2780022514 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:30.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672357374 0 0) 0x7f25900b40c0 con 0x7f25900b1f60 2024-03-29T11:32:30.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c003520 con 0x7f25900b1f60 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822904067 0 0) 0x7f259006f260 con 0x7f2590143d10 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.601+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25900b40c0 con 0x7f2590143d10 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926905009 0 0) 0x7f257c003520 con 0x7f25900b1f60 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 >> v1:172.21.15.136:6789/0 conn(0x7f2590143d10 legacy=0x7f2590146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 >> v1:172.21.15.64:6789/0 conn(0x7f2590071eb0 legacy=0x7f2590146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 1 monclient: found mon.noname-b 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2590155d70 con 0x7f25900b1f60 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2588001a10 con 0x7f25900b1f60 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f2587fff640 10 monclient: dump: 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f25979e8640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.602+0000 7f25979e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2588001b70 con 0x7f25900b1f60 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f2587fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f2587fff640 1 -- 172.21.15.64:0/2780022514 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2588001f80 con 0x7f25900b1f60 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f2587fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f2587fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f2587fff640 10 monclient: dump: 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f25979e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.604+0000 7f25979e8640 4 set_mon_vals no callback set 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.605+0000 7f25979e8640 10 monclient: shutdown 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.605+0000 7f25979e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.605+0000 7f25979e8640 1 -- 172.21.15.64:0/2780022514 >> v1:172.21.15.64:6790/0 conn(0x7f25900b1f60 legacy=0x7f2590143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.605+0000 7f25979e8640 1 -- 172.21.15.64:0/2780022514 shutdown_connections 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.605+0000 7f25979e8640 1 -- 172.21.15.64:0/2780022514 >> 172.21.15.64:0/2780022514 conn(0x7f2590066a00 msgr2=0x7f2590066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.606+0000 7f25979e8640 1 -- 172.21.15.64:0/2780022514 shutdown_connections 2024-03-29T11:32:30.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.606+0000 7f25979e8640 1 -- 172.21.15.64:0/2780022514 wait complete. 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.607+0000 7f25979e8640 10 monclient: build_initial_monmap 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.607+0000 7f25979e8640 10 monclient: monmap: 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.607+0000 7f25979e8640 1 librados: starting msgr at 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.607+0000 7f25979e8640 1 librados: starting objecter 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.608+0000 7f25979e8640 1 Processor -- start 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.608+0000 7f25979e8640 1 -- start start 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.608+0000 7f25979e8640 1 librados: setting wanted keys 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.608+0000 7f25979e8640 1 librados: calling monclient init 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.608+0000 7f25979e8640 10 monclient: init 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.609+0000 7f25979e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.609+0000 7f25979e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.609+0000 7f25979e8640 10 monclient(hunting): picked mon.noname-b con 0x7f2590143d10 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.610+0000 7f2595784640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2590143d10 0x7f25901df390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:34550/0 (socket says 172.21.15.64:34550) 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 10 monclient(hunting): picked mon.noname-c con 0x7f2590154c20 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f2594f83640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2590154c20 0x7f2590152020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50084/0 (socket says 172.21.15.64:50084) 2024-03-29T11:32:30.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f2594f83640 1 -- 172.21.15.64:0/2289940068 learned_addr learned my addr 172.21.15.64:0/2289940068 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 10 monclient(hunting): picked mon.noname-a con 0x7f2590152730 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259014b340 con 0x7f2590152730 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25901dfd40 con 0x7f2590143d10 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2590152b60 con 0x7f2590154c20 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.611+0000 7f25979e8640 10 monclient(hunting): authenticate will time out at 1686.881322s 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 client.?.objecter ms_handle_connect 0x7f2590143d10 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 client.?.objecter ms_handle_connect 0x7f2590154c20 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876505075 0 0) 0x7f25901dfd40 con 0x7f2590143d10 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2574004eb0 con 0x7f2590143d10 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684755961 0 0) 0x7f2590152b60 con 0x7f2590154c20 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.612+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25901dfd40 con 0x7f2590154c20 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 client.?.objecter ms_handle_connect 0x7f2590152730 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127653921 0 0) 0x7f2574004eb0 con 0x7f2590143d10 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2590152b60 con 0x7f2590143d10 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2588001f40 con 0x7f2590143d10 2024-03-29T11:32:30.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587268267 0 0) 0x7f25901dfd40 con 0x7f2590154c20 2024-03-29T11:32:30.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2574005990 con 0x7f2590154c20 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2580001970 con 0x7f2590154c20 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.613+0000 7f2585ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:30.619 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1462318871 0 0) 0x7f2590152b60 con 0x7f2590143d10 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 >> v1:172.21.15.64:6790/0 conn(0x7f2590154c20 legacy=0x7f2590152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 >> v1:172.21.15.64:6789/0 conn(0x7f2590152730 legacy=0x7f25901479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 monclient: found mon.b 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259014b590 con 0x7f2590143d10 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 5 monclient: authenticate success, global_id 4561 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 10 monclient: _renew_subs 2024-03-29T11:32:30.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25901480e0 con 0x7f2590143d10 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 10 monclient: _renew_subs 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2590148610 con 0x7f2590143d10 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25979e8640 1 librados: init done 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2588004e10 con 0x7f2590143d10 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2588001f40 con 0x7f2590143d10 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f25877fe640 4 set_mon_vals no callback set 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.614+0000 7f2585ffb640 10 monclient: dump: 2024-03-29T11:32:30.621 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.616+0000 7f25979e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.616+0000 7f25979e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.616+0000 7f25979e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.616+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2570005080 con 0x7f2590143d10 2024-03-29T11:32:30.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.617+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f258801adf0 con 0x7f2590143d10 2024-03-29T11:32:30.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.618+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f2588054970 con 0x7f2590143d10 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.618+0000 7f2585ffb640 10 client.4561.objecter ms_dispatch 0x7f2590000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.618+0000 7f2585ffb640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.618+0000 7f2585ffb640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.618+0000 7f2585ffb640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.619+0000 7f2594f83640 10 monclient: get_auth_request con 0x7f257406e280 auth_method 0 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.620+0000 7f2585ffb640 10 client.4561.objecter ms_handle_connect 0x7f257406e280 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.625+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2588050c10 con 0x7f2590143d10 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.625+0000 7f2585ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:30.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.625+0000 7f2585ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:30.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.996+0000 7f25979e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:30.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.996+0000 7f25979e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:30.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.996+0000 7f25979e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:32:30.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.996+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2570005330 con 0x7f2590143d10 2024-03-29T11:32:30.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.997+0000 7f2585ffb640 1 -- 172.21.15.64:0/2289940068 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f2588050c10 con 0x7f2590143d10 2024-03-29T11:32:30.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.997+0000 7f2585ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:30.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:30.997+0000 7f2585ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:30.994 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:30.994 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:30.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.003+0000 7f25979e8640 10 librados: watch_flush enter 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.003+0000 7f25979e8640 10 librados: watch_flush exit 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.003+0000 7f25979e8640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.003+0000 7f25979e8640 10 client.4561.objecter successfully canceled tick 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 >> v1:172.21.15.136:6816/2312274743 conn(0x7f257406e280 legacy=0x7f2574070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 10 monclient: shutdown 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:30.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 >> v1:172.21.15.136:6789/0 conn(0x7f2590143d10 legacy=0x7f25901df390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:31.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 shutdown_connections 2024-03-29T11:32:31.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 >> 172.21.15.64:0/2289940068 conn(0x7f2590066a00 msgr2=0x7f25900b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:31.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 shutdown_connections 2024-03-29T11:32:31.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 -- 172.21.15.64:0/2289940068 wait complete. 2024-03-29T11:32:31.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:31.004+0000 7f25979e8640 1 librados: shutdown 2024-03-29T11:32:31.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:37.825 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:37.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.978+0000 7ff05d34d640 10 monclient: get_monmap_and_config 2024-03-29T11:32:37.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.978+0000 7ff05d34d640 10 monclient: build_initial_monmap 2024-03-29T11:32:37.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.978+0000 7ff05d34d640 10 monclient: monmap: 2024-03-29T11:32:37.974 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:37.974 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:37.974 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:37.979746+0000 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:37.979746+0000 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:37.975 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.979+0000 7ff05d34d640 1 Processor -- start 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.979+0000 7ff05d34d640 1 -- start start 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.979+0000 7ff05d34d640 10 monclient: init 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0580a9750 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient(hunting): picked mon.noname-b con 0x7ff05813e1e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient(hunting): picked mon.noname-c con 0x7ff058071c30 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0580ab300 con 0x7ff0580a9750 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0580ac2f0 con 0x7ff058071c30 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0580ad8d0 con 0x7ff05813e1e0 2024-03-29T11:32:37.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:37.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff05d34d640 10 monclient(hunting): authenticate will time out at 1694.251204s 2024-03-29T11:32:37.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff0577fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff058071c30 0x7ff0580ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:51510/0 (socket says 172.21.15.64:51510) 2024-03-29T11:32:37.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.981+0000 7ff0577fe640 1 -- 172.21.15.64:0/566933223 learned_addr learned my addr 172.21.15.64:0/566933223 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708269306 0 0) 0x7ff0580ac2f0 con 0x7ff058071c30 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff040003520 con 0x7ff058071c30 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708165295 0 0) 0x7ff0580ad8d0 con 0x7ff05813e1e0 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0580ac2f0 con 0x7ff05813e1e0 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325077415 0 0) 0x7ff0580ab300 con 0x7ff0580a9750 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.982+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0580ad8d0 con 0x7ff0580a9750 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 996947786 0 0) 0x7ff040003520 con 0x7ff058071c30 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 >> v1:172.21.15.64:6790/0 conn(0x7ff05813e1e0 legacy=0x7ff0580ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 >> v1:172.21.15.64:6789/0 conn(0x7ff0580a9750 legacy=0x7ff05813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 monclient: found mon.noname-c 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:37.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0580ada60 con 0x7ff058071c30 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff050001a10 con 0x7ff058071c30 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: dump: 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.983 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff05d34d640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff05d34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff050001b70 con 0x7ff058071c30 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 1 -- 172.21.15.64:0/566933223 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff050001f80 con 0x7ff058071c30 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff055ffb640 10 monclient: dump: 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:37.984 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff05d34d640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.983+0000 7ff05d34d640 4 set_mon_vals no callback set 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 10 monclient: shutdown 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 1 -- 172.21.15.64:0/566933223 >> v1:172.21.15.136:6789/0 conn(0x7ff058071c30 legacy=0x7ff0580ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 1 -- 172.21.15.64:0/566933223 shutdown_connections 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 1 -- 172.21.15.64:0/566933223 >> 172.21.15.64:0/566933223 conn(0x7ff05806b660 msgr2=0x7ff05806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 1 -- 172.21.15.64:0/566933223 shutdown_connections 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.984+0000 7ff05d34d640 1 -- 172.21.15.64:0/566933223 wait complete. 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.985+0000 7ff05d34d640 10 monclient: build_initial_monmap 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.986+0000 7ff05d34d640 10 monclient: monmap: 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:37.986 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.986+0000 7ff05d34d640 1 librados: starting msgr at 2024-03-29T11:32:37.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.986+0000 7ff05d34d640 1 librados: starting objecter 2024-03-29T11:32:37.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.987+0000 7ff05d34d640 1 Processor -- start 2024-03-29T11:32:37.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.987+0000 7ff05d34d640 1 -- start start 2024-03-29T11:32:37.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.987+0000 7ff05d34d640 1 librados: setting wanted keys 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.987+0000 7ff05d34d640 1 librados: calling monclient init 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.987+0000 7ff05d34d640 10 monclient: init 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.988+0000 7ff05d34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.988+0000 7ff05d34d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.988+0000 7ff05d34d640 10 monclient(hunting): picked mon.noname-a con 0x7ff058071c30 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.988+0000 7ff05d34d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0580a9750 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.988+0000 7ff0567fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff0580a9750 0x7ff05811de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:52620/0 (socket says 172.21.15.64:52620) 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff0577fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff05813e1e0 0x7ff058070d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:44974/0 (socket says 172.21.15.64:44974) 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff056ffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff058071c30 0x7ff05811f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37440/0 (socket says 172.21.15.64:37440) 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff0567fc640 1 -- 172.21.15.64:0/3972752445 learned_addr learned my addr 172.21.15.64:0/3972752445 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff05d34d640 10 monclient(hunting): picked mon.noname-c con 0x7ff05813e1e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff03f7fe640 10 client.?.objecter ms_handle_connect 0x7ff0580a9750 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff03f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:37.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05811f230 con 0x7ff058071c30 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff03f7fe640 10 client.?.objecter ms_handle_connect 0x7ff05813e1e0 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff03f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05811f510 con 0x7ff0580a9750 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff03f7fe640 10 client.?.objecter ms_handle_connect 0x7ff058071c30 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff03f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.989+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff058150eb0 con 0x7ff05813e1e0 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff05d34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff05d34d640 10 monclient(hunting): authenticate will time out at 1694.259879s 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310190524 0 0) 0x7ff05811f510 con 0x7ff0580a9750 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:32:37.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff044004eb0 con 0x7ff0580a9750 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070374050 0 0) 0x7ff058150eb0 con 0x7ff05813e1e0 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05811f510 con 0x7ff05813e1e0 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173718907 0 0) 0x7ff05811f230 con 0x7ff058071c30 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.990+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff058150eb0 con 0x7ff058071c30 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619394029 0 0) 0x7ff05811f510 con 0x7ff05813e1e0 2024-03-29T11:32:37.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff044006680 con 0x7ff05813e1e0 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff050001e80 con 0x7ff05813e1e0 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976092124 0 0) 0x7ff044004eb0 con 0x7ff0580a9750 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff05811f510 con 0x7ff0580a9750 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff04c001910 con 0x7ff0580a9750 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:37.997 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4166135884 0 0) 0x7ff058150eb0 con 0x7ff058071c30 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff044004eb0 con 0x7ff058071c30 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff048001870 con 0x7ff058071c30 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.991+0000 7ff03f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:37.998 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2953606223 0 0) 0x7ff044006680 con 0x7ff05813e1e0 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 >> v1:172.21.15.136:6789/0 conn(0x7ff0580a9750 legacy=0x7ff05811de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 >> v1:172.21.15.64:6789/0 conn(0x7ff058071c30 legacy=0x7ff05811f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 monclient: found mon.c 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058151040 con 0x7ff05813e1e0 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:37.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 10 monclient: _renew_subs 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05811ceb0 con 0x7ff05813e1e0 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 10 monclient: _renew_subs 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0581e3590 con 0x7ff05813e1e0 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff050001420 con 0x7ff05813e1e0 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ff050004fb0 con 0x7ff05813e1e0 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff03f7fe640 10 monclient: dump: 2024-03-29T11:32:38.000 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff0557fa640 4 set_mon_vals no callback set 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.992+0000 7ff05d34d640 1 librados: init done 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.993+0000 7ff05d34d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.994+0000 7ff05d34d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.994+0000 7ff05d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.994+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0580abb00 con 0x7ff05813e1e0 2024-03-29T11:32:38.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.994+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7ff05001af80 con 0x7ff05813e1e0 2024-03-29T11:32:38.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.995+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7ff050084d70 con 0x7ff05813e1e0 2024-03-29T11:32:38.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.995+0000 7ff03f7fe640 10 client.4556.objecter ms_dispatch 0x7ff058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:38.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.995+0000 7ff03f7fe640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:38.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.995+0000 7ff03f7fe640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:38.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.995+0000 7ff03f7fe640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:32:38.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.996+0000 7ff056ffd640 10 monclient: get_auth_request con 0x7ff04406e4c0 auth_method 0 2024-03-29T11:32:38.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:37.998+0000 7ff03f7fe640 10 client.4556.objecter ms_handle_connect 0x7ff04406e4c0 2024-03-29T11:32:38.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.004+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff050050480 con 0x7ff05813e1e0 2024-03-29T11:32:38.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.005+0000 7ff03f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:38.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.005+0000 7ff03f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:38.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.388+0000 7ff05d34d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:38.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.388+0000 7ff05d34d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:38.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.389+0000 7ff05d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:38.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.389+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff058075f10 con 0x7ff05813e1e0 2024-03-29T11:32:38.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.390+0000 7ff03f7fe640 1 -- 172.21.15.64:0/3972752445 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7ff050001590 con 0x7ff05813e1e0 2024-03-29T11:32:38.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.390+0000 7ff03f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:38.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.390+0000 7ff03f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:38.385 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:38.385 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:38.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.393+0000 7ff05d34d640 10 librados: watch_flush enter 2024-03-29T11:32:38.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.393+0000 7ff05d34d640 10 librados: watch_flush exit 2024-03-29T11:32:38.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.393+0000 7ff05d34d640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:32:38.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.393+0000 7ff05d34d640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:32:38.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.393+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 >> v1:172.21.15.136:6816/2312274743 conn(0x7ff04406e4c0 legacy=0x7ff044070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 10 monclient: shutdown 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 >> v1:172.21.15.64:6790/0 conn(0x7ff05813e1e0 legacy=0x7ff058070d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 shutdown_connections 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 >> 172.21.15.64:0/3972752445 conn(0x7ff05806b660 msgr2=0x7ff0580b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 shutdown_connections 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 1 -- 172.21.15.64:0/3972752445 wait complete. 2024-03-29T11:32:38.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:38.394+0000 7ff05d34d640 1 librados: shutdown 2024-03-29T11:32:38.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:45.212 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:45.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.375+0000 7f7929948640 10 monclient: get_monmap_and_config 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.375+0000 7f7929948640 10 monclient: build_initial_monmap 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.375+0000 7f7929948640 10 monclient: monmap: 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:45.376166+0000 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:45.376166+0000 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:45.371 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:45.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.376+0000 7f7929948640 1 Processor -- start 2024-03-29T11:32:45.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.376+0000 7f7929948640 1 -- start start 2024-03-29T11:32:45.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.376+0000 7f7929948640 10 monclient: init 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient(hunting): picked mon.noname-a con 0x7f79240b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient(hunting): picked mon.noname-b con 0x7f792414a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient(hunting): picked mon.noname-c con 0x7f79240ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79240aba70 con 0x7f79240b0770 2024-03-29T11:32:45.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79240aecb0 con 0x7f79240ac5a0 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79240af150 con 0x7f792414a700 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.377+0000 7f7929948640 10 monclient(hunting): authenticate will time out at 1701.647113s 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7922ffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f79240b0770 0x7f7924149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48797/0 (socket says 172.21.15.64:48797) 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7922ffd640 1 -- 172.21.15.64:0/2323540760 learned_addr learned my addr 172.21.15.64:0/2323540760 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541868426 0 0) 0x7f79240aba70 con 0x7f79240b0770 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f790c003520 con 0x7f79240b0770 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154764951 0 0) 0x7f79240aecb0 con 0x7f79240ac5a0 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:45.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:45.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:45.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:32:45.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.380+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79240aba70 con 0x7f79240ac5a0 2024-03-29T11:32:45.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1187926181 0 0) 0x7f79240aba70 con 0x7f79240ac5a0 2024-03-29T11:32:45.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 >> v1:172.21.15.64:6790/0 conn(0x7f792414a700 legacy=0x7f79240abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:45.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 >> v1:172.21.15.64:6789/0 conn(0x7f79240b0770 legacy=0x7f7924149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:45.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 1 monclient: found mon.noname-c 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79240ac9d0 con 0x7f79240ac5a0 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7914001670 con 0x7f79240ac5a0 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7921ffb640 10 monclient: dump: 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:45.382 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7929948640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.383+0000 7f7929948640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79140017d0 con 0x7f79240ac5a0 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7921ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7921ffb640 1 -- 172.21.15.64:0/2323540760 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7914001ba0 con 0x7f79240ac5a0 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7921ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:45.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7921ffb640 10 monclient: dump: 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7929948640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.384+0000 7f7929948640 4 set_mon_vals no callback set 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.385+0000 7f7929948640 10 monclient: shutdown 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.385+0000 7f7929948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:45.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.385+0000 7f7929948640 1 -- 172.21.15.64:0/2323540760 >> v1:172.21.15.136:6789/0 conn(0x7f79240ac5a0 legacy=0x7f7924151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.385+0000 7f7929948640 1 -- 172.21.15.64:0/2323540760 shutdown_connections 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.385+0000 7f7929948640 1 -- 172.21.15.64:0/2323540760 >> 172.21.15.64:0/2323540760 conn(0x7f792406b660 msgr2=0x7f792406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.386+0000 7f7929948640 1 -- 172.21.15.64:0/2323540760 shutdown_connections 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.386+0000 7f7929948640 1 -- 172.21.15.64:0/2323540760 wait complete. 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.387+0000 7f7929948640 10 monclient: build_initial_monmap 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.387+0000 7f7929948640 10 monclient: monmap: 2024-03-29T11:32:45.385 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.387+0000 7f7929948640 1 librados: starting msgr at 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.387+0000 7f7929948640 1 librados: starting objecter 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.387+0000 7f7929948640 1 Processor -- start 2024-03-29T11:32:45.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 1 -- start start 2024-03-29T11:32:45.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 1 librados: setting wanted keys 2024-03-29T11:32:45.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 1 librados: calling monclient init 2024-03-29T11:32:45.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 10 monclient: init 2024-03-29T11:32:45.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:45.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:45.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 10 monclient(hunting): picked mon.noname-b con 0x7f79240ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.388+0000 7f7929948640 10 monclient(hunting): picked mon.noname-a con 0x7f79240b0770 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.389+0000 7f7929948640 10 monclient(hunting): picked mon.noname-c con 0x7f792414a700 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.389+0000 7f7929948640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79241efe50 con 0x7f79240b0770 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.389+0000 7f7929948640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79241efb00 con 0x7f79240ac5a0 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.389+0000 7f7929948640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792407e5e0 con 0x7f792414a700 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.389+0000 7f7929948640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.389+0000 7f7929948640 10 monclient(hunting): authenticate will time out at 1701.658904s 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.392+0000 7f79227fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f79240b0770 0x7f792407ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48959/0 (socket says 172.21.15.64:48959) 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.392+0000 7f79227fc640 1 -- 172.21.15.64:0/1845402482 learned_addr learned my addr 172.21.15.64:0/1845402482 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 10 client.?.objecter ms_handle_connect 0x7f79240b0770 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:45.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814274581 0 0) 0x7f79241efe50 con 0x7f79240b0770 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7910004eb0 con 0x7f79240b0770 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.395+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190538136 0 0) 0x7f7910004eb0 con 0x7f79240b0770 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79241efe50 con 0x7f79240b0770 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f79180017d0 con 0x7f79240b0770 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient(hunting): dump: 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:45.398 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 client.?.objecter ms_handle_connect 0x7f79240ac5a0 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3932925565 0 0) 0x7f79241efe50 con 0x7f79240b0770 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 >> v1:172.21.15.64:6790/0 conn(0x7f792414a700 legacy=0x7f79241f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 >> v1:172.21.15.136:6789/0 conn(0x7f79240ac5a0 legacy=0x7f79241f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 monclient: found mon.a 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792407d300 con 0x7f79240b0770 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient: _finish_auth 0 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f790bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f7929948640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f7929948640 10 monclient: _renew_subs 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f7929948640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.396+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792407d550 con 0x7f79240b0770 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f7929948640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f7929948640 10 monclient: _renew_subs 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f7929948640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:32:45.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f792407b1d0 con 0x7f79240b0770 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f7929948640 1 librados: init done 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7918001930 con 0x7f79240b0770 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f790bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7918004a40 con 0x7f79240b0770 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f790bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f790bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f790bfff640 10 monclient: dump: 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:45.401 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.397+0000 7f79217fa640 4 set_mon_vals no callback set 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.399+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f7918004ba0 con 0x7f79240b0770 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f7918084ac0 con 0x7f79240b0770 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f790bfff640 10 client.4524.objecter ms_dispatch 0x7f7924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f790bfff640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f790bfff640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f790bfff640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f7922ffd640 10 monclient: get_auth_request con 0x7f791006dca0 auth_method 0 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f7929948640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f7929948640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:45.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f7929948640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:32:45.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.400+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904005080 con 0x7f79240b0770 2024-03-29T11:32:45.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.402+0000 7f790bfff640 10 client.4524.objecter ms_handle_connect 0x7f791006dca0 2024-03-29T11:32:45.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.410+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79180501d0 con 0x7f79240b0770 2024-03-29T11:32:45.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.410+0000 7f790bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:45.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.410+0000 7f790bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:45.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.760+0000 7f7929948640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:45.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.760+0000 7f7929948640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:45.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.760+0000 7f7929948640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:32:45.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.760+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7904005330 con 0x7f79240b0770 2024-03-29T11:32:45.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.761+0000 7f790bfff640 1 -- 172.21.15.64:0/1845402482 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f79180522d0 con 0x7f79240b0770 2024-03-29T11:32:45.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.761+0000 7f790bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:45.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.761+0000 7f790bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:45.757 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:45.757 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:45.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 10 librados: watch_flush enter 2024-03-29T11:32:45.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 10 librados: watch_flush exit 2024-03-29T11:32:45.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:32:45.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:32:45.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 >> v1:172.21.15.136:6816/2312274743 conn(0x7f791006dca0 legacy=0x7f7910070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 10 monclient: shutdown 2024-03-29T11:32:45.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:45.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 >> v1:172.21.15.64:6789/0 conn(0x7f79240b0770 legacy=0x7f792407ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 shutdown_connections 2024-03-29T11:32:45.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.765+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 >> 172.21.15.64:0/1845402482 conn(0x7f792406b660 msgr2=0x7f79240b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:45.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.766+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 shutdown_connections 2024-03-29T11:32:45.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.766+0000 7f7929948640 1 -- 172.21.15.64:0/1845402482 wait complete. 2024-03-29T11:32:45.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:45.766+0000 7f7929948640 1 librados: shutdown 2024-03-29T11:32:45.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:52.584 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:52.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.739+0000 7f582c2d6640 10 monclient: get_monmap_and_config 2024-03-29T11:32:52.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.739+0000 7f582c2d6640 10 monclient: build_initial_monmap 2024-03-29T11:32:52.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.739+0000 7f582c2d6640 10 monclient: monmap: 2024-03-29T11:32:52.736 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:52.736 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:52.736 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:32:52.740562+0000 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:32:52.740562+0000 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.740+0000 7f582c2d6640 1 Processor -- start 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.740+0000 7f582c2d6640 1 -- start start 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.740+0000 7f582c2d6640 10 monclient: init 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.741+0000 7f582c2d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.741+0000 7f582c2d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 10 monclient(hunting): picked mon.noname-b con 0x7f58240a9ac0 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 10 monclient(hunting): picked mon.noname-a con 0x7f582413ee90 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 10 monclient(hunting): picked mon.noname-c con 0x7f58240af900 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58240afd10 con 0x7f582413ee90 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58240af6f0 con 0x7f58240af900 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58240aac80 con 0x7f58240a9ac0 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:52.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.742+0000 7f582c2d6640 10 monclient(hunting): authenticate will time out at 1709.011959s 2024-03-29T11:32:52.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.743+0000 7f582a072640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f58240a9ac0 0x7f58240adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56322/0 (socket says 172.21.15.64:56322) 2024-03-29T11:32:52.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.743+0000 7f582a072640 1 -- 172.21.15.64:0/2692429875 learned_addr learned my addr 172.21.15.64:0/2692429875 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.746+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049137196 0 0) 0x7f58240aac80 con 0x7f58240a9ac0 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.746+0000 7f5829070640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.746+0000 7f5829070640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.746+0000 7f5829070640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.746+0000 7f5829070640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.746+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5810003db0 con 0x7f58240a9ac0 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2614594025 0 0) 0x7f5810003db0 con 0x7f58240a9ac0 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 >> v1:172.21.15.136:6789/0 conn(0x7f58240af900 legacy=0x7f5824149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 >> v1:172.21.15.64:6789/0 conn(0x7f582413ee90 legacy=0x7f582413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:52.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: _finish_hunting 0 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 1 monclient: found mon.noname-b 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58240aaf60 con 0x7f58240a9ac0 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: _finish_auth 0 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: _check_auth_tickets 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5820001690 con 0x7f58240a9ac0 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f5829070640 10 monclient: dump: 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.744 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f582c2d6640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:32:52.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.747+0000 7f582c2d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:52.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58200018f0 con 0x7f58240a9ac0 2024-03-29T11:32:52.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f5829070640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:52.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f5829070640 1 -- 172.21.15.64:0/2692429875 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5820001d40 con 0x7f58240a9ac0 2024-03-29T11:32:52.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f5829070640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f5829070640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f5829070640 10 monclient: dump: 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:52.747 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f582c2d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.748+0000 7f582c2d6640 4 set_mon_vals no callback set 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.749+0000 7f582c2d6640 10 monclient: shutdown 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.749+0000 7f582c2d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.749+0000 7f582c2d6640 1 -- 172.21.15.64:0/2692429875 >> v1:172.21.15.64:6790/0 conn(0x7f58240a9ac0 legacy=0x7f58240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.750+0000 7f582c2d6640 1 -- 172.21.15.64:0/2692429875 shutdown_connections 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.750+0000 7f582c2d6640 1 -- 172.21.15.64:0/2692429875 >> 172.21.15.64:0/2692429875 conn(0x7f58240b35e0 msgr2=0x7f58240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.750+0000 7f582c2d6640 1 -- 172.21.15.64:0/2692429875 shutdown_connections 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.750+0000 7f582c2d6640 1 -- 172.21.15.64:0/2692429875 wait complete. 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.751+0000 7f582c2d6640 10 monclient: build_initial_monmap 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.751+0000 7f582c2d6640 10 monclient: monmap: 2024-03-29T11:32:52.748 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:32:52.749 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.751+0000 7f582c2d6640 1 librados: starting msgr at 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.751+0000 7f582c2d6640 1 librados: starting objecter 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.752+0000 7f582c2d6640 1 Processor -- start 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.752+0000 7f582c2d6640 1 -- start start 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.752+0000 7f582c2d6640 1 librados: setting wanted keys 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.752+0000 7f582c2d6640 1 librados: calling monclient init 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.752+0000 7f582c2d6640 10 monclient: init 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:52.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient(hunting): picked mon.noname-c con 0x7f58240a9ac0 addr v1:172.21.15.64:6790/0 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient(hunting): picked mon.noname-b con 0x7f58240af900 addr v1:172.21.15.136:6789/0 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient(hunting): picked mon.noname-a con 0x7f582413ee90 addr v1:172.21.15.64:6789/0 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582412d480 con 0x7f582413ee90 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5824137170 con 0x7f58240af900 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5824141860 con 0x7f58240a9ac0 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:52.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.753+0000 7f582c2d6640 10 monclient(hunting): authenticate will time out at 1709.023471s 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f582a072640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f58240a9ac0 0x7f58241402f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58042/0 (socket says 172.21.15.64:58042) 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f582a072640 1 -- 172.21.15.64:0/3598532765 learned_addr learned my addr 172.21.15.64:0/3598532765 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f581affd640 10 client.?.objecter ms_handle_connect 0x7f58240a9ac0 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f581affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f581affd640 10 client.?.objecter ms_handle_connect 0x7f58240af900 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f581affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.756+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688519642 0 0) 0x7f5824137170 con 0x7f58240af900 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5808004eb0 con 0x7f58240af900 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721493935 0 0) 0x7f5824141860 con 0x7f58240a9ac0 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:32:52.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.757+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5808006230 con 0x7f58240a9ac0 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591483450 0 0) 0x7f5808006230 con 0x7f58240a9ac0 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58080066a0 con 0x7f58240a9ac0 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5820001650 con 0x7f58240a9ac0 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): dump: 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.756 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 client.?.objecter ms_handle_connect 0x7f582413ee90 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1151804551 0 0) 0x7f5808004eb0 con 0x7f58240af900 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5808006230 con 0x7f58240af900 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5814001830 con 0x7f58240af900 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): dump: 2024-03-29T11:32:52.757 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567347918 0 0) 0x7f582412d480 con 0x7f582413ee90 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.759+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5808004eb0 con 0x7f582413ee90 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1540986475 0 0) 0x7f58080066a0 con 0x7f58240a9ac0 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 >> v1:172.21.15.136:6789/0 conn(0x7f58240af900 legacy=0x7f582413d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 >> v1:172.21.15.64:6789/0 conn(0x7f582413ee90 legacy=0x7f58241e6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 1 monclient: found mon.c 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5824141b40 con 0x7f58240a9ac0 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 10 monclient: _finish_auth 0 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f581affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 10 monclient: _renew_subs 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:52.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5824141e20 con 0x7f58240a9ac0 2024-03-29T11:32:52.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:52.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 10 monclient: _renew_subs 2024-03-29T11:32:52.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:52.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5824142240 con 0x7f58240a9ac0 2024-03-29T11:32:52.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.760+0000 7f582c2d6640 1 librados: init done 2024-03-29T11:32:52.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5820004a90 con 0x7f58240a9ac0 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f581affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f5820004da0 con 0x7f58240a9ac0 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f581affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f581affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f581affd640 10 monclient: dump: 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:32:52.762 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.761+0000 7f582886f640 4 set_mon_vals no callback set 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.763+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f582001ad70 con 0x7f58240a9ac0 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.763+0000 7f582c2d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.763+0000 7f582c2d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.763+0000 7f582c2d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.763+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c005080 con 0x7f58240a9ac0 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.764+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f5820054910 con 0x7f58240a9ac0 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.764+0000 7f581affd640 10 client.4562.objecter ms_dispatch 0x7f5824000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.764+0000 7f581affd640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.764+0000 7f581affd640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.764+0000 7f581affd640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.764+0000 7f5829871640 10 monclient: get_auth_request con 0x7f580806e250 auth_method 0 2024-03-29T11:32:52.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.765+0000 7f581affd640 10 client.4562.objecter ms_handle_connect 0x7f580806e250 2024-03-29T11:32:52.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.776+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5820050bb0 con 0x7f58240a9ac0 2024-03-29T11:32:52.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.776+0000 7f581affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:52.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:52.776+0000 7f581affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:53.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.111+0000 7f582c2d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:53.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.111+0000 7f582c2d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:53.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.111+0000 7f582c2d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:32:53.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.111+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f580c005330 con 0x7f58240a9ac0 2024-03-29T11:32:53.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.112+0000 7f581affd640 1 -- 172.21.15.64:0/3598532765 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f5820050bb0 con 0x7f58240a9ac0 2024-03-29T11:32:53.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.112+0000 7f581affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:53.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.112+0000 7f581affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:53.109 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:32:53.109 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:53.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.115+0000 7f582c2d6640 10 librados: watch_flush enter 2024-03-29T11:32:53.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.115+0000 7f582c2d6640 10 librados: watch_flush exit 2024-03-29T11:32:53.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.115+0000 7f582c2d6640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:32:53.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.115+0000 7f582c2d6640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:32:53.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.117+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 >> v1:172.21.15.136:6816/2312274743 conn(0x7f580806e250 legacy=0x7f5808070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.117+0000 7f582c2d6640 10 monclient: shutdown 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.117+0000 7f582c2d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.117+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 >> v1:172.21.15.64:6790/0 conn(0x7f58240a9ac0 legacy=0x7f58241402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.117+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 shutdown_connections 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.117+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 >> 172.21.15.64:0/3598532765 conn(0x7f58240b35e0 msgr2=0x7f58240adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.118+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 shutdown_connections 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.118+0000 7f582c2d6640 1 -- 172.21.15.64:0/3598532765 wait complete. 2024-03-29T11:32:53.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:32:53.118+0000 7f582c2d6640 1 librados: shutdown 2024-03-29T11:32:53.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:59.936 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.077+0000 7fe37d640640 10 monclient: get_monmap_and_config 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.077+0000 7fe37d640640 10 monclient: build_initial_monmap 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.077+0000 7fe37d640640 10 monclient: monmap: 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:00.079115+0000 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:00.079115+0000 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:00.074 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:00.075 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:00.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.078+0000 7fe37d640640 1 Processor -- start 2024-03-29T11:33:00.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.078+0000 7fe37d640640 1 -- start start 2024-03-29T11:33:00.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.078+0000 7fe37d640640 10 monclient: init 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient(hunting): picked mon.noname-a con 0x7fe3781457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient(hunting): picked mon.noname-b con 0x7fe37813f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient(hunting): picked mon.noname-c con 0x7fe3780a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3780aaef0 con 0x7fe3781457f0 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe378145bd0 con 0x7fe3780a9d70 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe378145eb0 con 0x7fe37813f650 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:00.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe37d640640 10 monclient(hunting): authenticate will time out at 1716.350071s 2024-03-29T11:33:00.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe3767fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe37813f650 0x7fe3780b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58060/0 (socket says 172.21.15.64:58060) 2024-03-29T11:33:00.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe3767fc640 1 -- 172.21.15.64:0/2233028048 learned_addr learned my addr 172.21.15.64:0/2233028048 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:00.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.079+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867405706 0 0) 0x7fe378145eb0 con 0x7fe37813f650 2024-03-29T11:33:00.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.080+0000 7fe375ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.080+0000 7fe375ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.080+0000 7fe375ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.080+0000 7fe375ffb640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.080+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe360003520 con 0x7fe37813f650 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.080+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276084042 0 0) 0x7fe360003520 con 0x7fe37813f650 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.081+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 >> v1:172.21.15.136:6789/0 conn(0x7fe3780a9d70 legacy=0x7fe37814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:00.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.081+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 >> v1:172.21.15.64:6789/0 conn(0x7fe3781457f0 legacy=0x7fe37813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 1 monclient: found mon.noname-b 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378146340 con 0x7fe37813f650 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe36c001690 con 0x7fe37813f650 2024-03-29T11:33:00.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.083+0000 7fe375ffb640 10 monclient: dump: 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:00.081 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.084+0000 7fe37d640640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.084+0000 7fe37d640640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.084+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe36c001860 con 0x7fe37813f650 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe375ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe375ffb640 1 -- 172.21.15.64:0/2233028048 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe36c001c70 con 0x7fe37813f650 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe375ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe375ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe375ffb640 10 monclient: dump: 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:00.082 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:00.083 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe37d640640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe37d640640 4 set_mon_vals no callback set 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe37d640640 10 monclient: shutdown 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe37d640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.085+0000 7fe37d640640 1 -- 172.21.15.64:0/2233028048 >> v1:172.21.15.64:6790/0 conn(0x7fe37813f650 legacy=0x7fe3780b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.086+0000 7fe37d640640 1 -- 172.21.15.64:0/2233028048 shutdown_connections 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.086+0000 7fe37d640640 1 -- 172.21.15.64:0/2233028048 >> 172.21.15.64:0/2233028048 conn(0x7fe378066a00 msgr2=0x7fe378066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.087+0000 7fe37d640640 1 -- 172.21.15.64:0/2233028048 shutdown_connections 2024-03-29T11:33:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.087+0000 7fe37d640640 1 -- 172.21.15.64:0/2233028048 wait complete. 2024-03-29T11:33:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.088+0000 7fe37d640640 10 monclient: build_initial_monmap 2024-03-29T11:33:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.088+0000 7fe37d640640 10 monclient: monmap: 2024-03-29T11:33:00.085 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:00.085 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:00.085 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 1 librados: starting msgr at 2024-03-29T11:33:00.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 1 librados: starting objecter 2024-03-29T11:33:00.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 1 Processor -- start 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 1 -- start start 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 1 librados: setting wanted keys 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 1 librados: calling monclient init 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.089+0000 7fe37d640640 10 monclient: init 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient(hunting): picked mon.noname-b con 0x7fe3780a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient(hunting): picked mon.noname-a con 0x7fe37813f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient(hunting): picked mon.noname-c con 0x7fe3781457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe378071580 con 0x7fe37813f650 2024-03-29T11:33:00.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe378070bc0 con 0x7fe3780a9d70 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37806fb60 con 0x7fe3781457f0 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.090+0000 7fe37d640640 10 monclient(hunting): authenticate will time out at 1716.361192s 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.091+0000 7fe3777fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe3781457f0 0x7fe3781e5c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33974/0 (socket says 172.21.15.64:33974) 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.091+0000 7fe3777fe640 1 -- 172.21.15.64:0/1285370109 learned_addr learned my addr 172.21.15.64:0/1285370109 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.092+0000 7fe35f7fe640 10 client.?.objecter ms_handle_connect 0x7fe3781457f0 2024-03-29T11:33:00.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.092+0000 7fe35f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:00.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.093+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956139084 0 0) 0x7fe37806fb60 con 0x7fe3781457f0 2024-03-29T11:33:00.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.093+0000 7fe35f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:00.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.093+0000 7fe35f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:00.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.093+0000 7fe35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:00.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.093+0000 7fe35f7fe640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:33:00.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.093+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe364004eb0 con 0x7fe3781457f0 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.095+0000 7fe35f7fe640 10 client.?.objecter ms_handle_connect 0x7fe3780a9d70 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.095+0000 7fe35f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.095+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927081985 0 0) 0x7fe364004eb0 con 0x7fe3781457f0 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.095+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe37806fb60 con 0x7fe3781457f0 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.095+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe370001b70 con 0x7fe3781457f0 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.095+0000 7fe35f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:00.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 monclient(hunting): dump: 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 client.?.objecter ms_handle_connect 0x7fe37813f650 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:00.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 325190833 0 0) 0x7fe37806fb60 con 0x7fe3781457f0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 >> v1:172.21.15.136:6789/0 conn(0x7fe3780a9d70 legacy=0x7fe378071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 >> v1:172.21.15.64:6789/0 conn(0x7fe37813f650 legacy=0x7fe37806f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 1 monclient: found mon.c 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.096+0000 7fe35f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37806fcf0 con 0x7fe3781457f0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe35f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe35f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe35f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 10 monclient: _renew_subs 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37806e840 con 0x7fe3781457f0 2024-03-29T11:33:00.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 10 monclient: _renew_subs 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3781e7510 con 0x7fe3781457f0 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe37d640640 1 librados: init done 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe370001cd0 con 0x7fe3781457f0 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe35f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.097+0000 7fe3757fa640 4 set_mon_vals no callback set 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.098+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fe370004e20 con 0x7fe3781457f0 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.098+0000 7fe35f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.099+0000 7fe35f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.099+0000 7fe35f7fe640 10 monclient: dump: 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:00.100 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.099+0000 7fe37d640640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.099+0000 7fe37d640640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.099+0000 7fe37d640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.099+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe358005080 con 0x7fe3781457f0 2024-03-29T11:33:00.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.100+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fe37001adf0 con 0x7fe3781457f0 2024-03-29T11:33:00.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.100+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fe370084de0 con 0x7fe3781457f0 2024-03-29T11:33:00.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.100+0000 7fe35f7fe640 10 client.4568.objecter ms_dispatch 0x7fe378000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:00.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.100+0000 7fe35f7fe640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:00.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.100+0000 7fe35f7fe640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:00.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.101+0000 7fe35f7fe640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:33:00.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.101+0000 7fe376ffd640 10 monclient: get_auth_request con 0x7fe36406daa0 auth_method 0 2024-03-29T11:33:00.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.101+0000 7fe35f7fe640 10 client.4568.objecter ms_handle_connect 0x7fe36406daa0 2024-03-29T11:33:00.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.108+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe370050720 con 0x7fe3781457f0 2024-03-29T11:33:00.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.108+0000 7fe35f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:00.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.108+0000 7fe35f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:00.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.444+0000 7fe37d640640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:00.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.444+0000 7fe37d640640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:00.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.444+0000 7fe37d640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:00.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.444+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe358005330 con 0x7fe3781457f0 2024-03-29T11:33:00.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.445+0000 7fe35f7fe640 1 -- 172.21.15.64:0/1285370109 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fe370052650 con 0x7fe3781457f0 2024-03-29T11:33:00.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.445+0000 7fe35f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:00.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.445+0000 7fe35f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:00.442 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:00.442 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.448+0000 7fe37d640640 10 librados: watch_flush enter 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 10 librados: watch_flush exit 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 >> v1:172.21.15.136:6816/2312274743 conn(0x7fe36406daa0 legacy=0x7fe36406ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 10 monclient: shutdown 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 >> v1:172.21.15.64:6790/0 conn(0x7fe3781457f0 legacy=0x7fe3781e5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 shutdown_connections 2024-03-29T11:33:00.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 >> 172.21.15.64:0/1285370109 conn(0x7fe378066a00 msgr2=0x7fe37813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:00.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 shutdown_connections 2024-03-29T11:33:00.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 -- 172.21.15.64:0/1285370109 wait complete. 2024-03-29T11:33:00.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:00.449+0000 7fe37d640640 1 librados: shutdown 2024-03-29T11:33:00.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:07.269 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:07.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.424+0000 7f7e13fff640 10 monclient: get_monmap_and_config 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.424+0000 7f7e13fff640 10 monclient: build_initial_monmap 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.424+0000 7f7e13fff640 10 monclient: monmap: 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:07.425739+0000 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:07.425739+0000 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:07.421 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:07.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.426+0000 7f7e13fff640 1 Processor -- start 2024-03-29T11:33:07.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.426+0000 7f7e13fff640 1 -- start start 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.426+0000 7f7e13fff640 10 monclient: init 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c0389e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0c038dc0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0c030660 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c0c54c0 con 0x7f7e0c030660 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c036430 con 0x7f7e0c0389e0 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c039840 con 0x7f7e0c038dc0 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:07.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.427+0000 7f7e13fff640 10 monclient(hunting): authenticate will time out at 1723.697180s 2024-03-29T11:33:07.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e127fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7e0c038dc0 0x7f7e0c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54889/0 (socket says 172.21.15.64:54889) 2024-03-29T11:33:07.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e127fc640 1 -- 172.21.15.64:0/2128895373 learned_addr learned my addr 172.21.15.64:0/2128895373 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051388240 0 0) 0x7f7e0c039840 con 0x7f7e0c038dc0 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e11ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.430+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dfc003520 con 0x7f7e0c038dc0 2024-03-29T11:33:07.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.431+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3744800201 0 0) 0x7f7dfc003520 con 0x7f7e0c038dc0 2024-03-29T11:33:07.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.431+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 >> v1:172.21.15.136:6789/0 conn(0x7f7e0c0389e0 legacy=0x7f7e0c033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:07.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.432+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 >> v1:172.21.15.64:6789/0 conn(0x7f7e0c030660 legacy=0x7f7e0c0d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:07.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:07.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 1 monclient: found mon.noname-b 2024-03-29T11:33:07.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:07.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:33:07.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c035690 con 0x7f7e0c038dc0 2024-03-29T11:33:07.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e13fff640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.434+0000 7f7e13fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7e1404f730 con 0x7f7e0c038dc0 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: dump: 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:07.432 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e13fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e140008d0 con 0x7f7e0c038dc0 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e13fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e13fff640 4 set_mon_vals no callback set 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 1 -- 172.21.15.64:0/2128895373 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7e140758b0 con 0x7f7e0c038dc0 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.435+0000 7f7e11ffb640 10 monclient: dump: 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:07.433 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:07.434 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:07.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 10 monclient: shutdown 2024-03-29T11:33:07.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:07.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 1 -- 172.21.15.64:0/2128895373 >> v1:172.21.15.64:6790/0 conn(0x7f7e0c038dc0 legacy=0x7f7e0c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 1 -- 172.21.15.64:0/2128895373 shutdown_connections 2024-03-29T11:33:07.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 1 -- 172.21.15.64:0/2128895373 >> 172.21.15.64:0/2128895373 conn(0x7f7e0c005c00 msgr2=0x7f7e0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 1 -- 172.21.15.64:0/2128895373 shutdown_connections 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.436+0000 7f7e13fff640 1 -- 172.21.15.64:0/2128895373 wait complete. 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.437+0000 7f7e13fff640 10 monclient: build_initial_monmap 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.437+0000 7f7e13fff640 10 monclient: monmap: 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.437+0000 7f7e13fff640 1 librados: starting msgr at 2024-03-29T11:33:07.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.437+0000 7f7e13fff640 1 librados: starting objecter 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.438+0000 7f7e13fff640 1 Processor -- start 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.438+0000 7f7e13fff640 1 -- start start 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.438+0000 7f7e13fff640 1 librados: setting wanted keys 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.438+0000 7f7e13fff640 1 librados: calling monclient init 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.438+0000 7f7e13fff640 10 monclient: init 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0c030660 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:07.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0c0389e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0c0c2ab0 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c009ea0 con 0x7f7e0c0c2ab0 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c00d7e0 con 0x7f7e0c030660 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c0c5650 con 0x7f7e0c0389e0 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.439+0000 7f7e13fff640 10 monclient(hunting): authenticate will time out at 1723.708983s 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.442+0000 7f7e137fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f7e0c0c2ab0 0x7f7e0c16c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45741/0 (socket says 172.21.15.64:45741) 2024-03-29T11:33:07.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.442+0000 7f7e137fe640 1 -- 172.21.15.64:0/4178881184 learned_addr learned my addr 172.21.15.64:0/4178881184 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:07.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.445+0000 7f7dfb7fe640 10 client.?.objecter ms_handle_connect 0x7f7e0c0c2ab0 2024-03-29T11:33:07.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 client.?.objecter ms_handle_connect 0x7f7e0c0389e0 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757737452 0 0) 0x7f7e0c0c5650 con 0x7f7e0c0389e0 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e00004eb0 con 0x7f7e0c0389e0 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.446+0000 7f7dfb7fe640 10 client.?.objecter ms_handle_connect 0x7f7e0c030660 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552729748 0 0) 0x7f7e0c009ea0 con 0x7f7e0c0c2ab0 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:07.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c0c5650 con 0x7f7e0c0c2ab0 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 782827784 0 0) 0x7f7e00004eb0 con 0x7f7e0c0389e0 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e0c009ea0 con 0x7f7e0c0389e0 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7e1404f400 con 0x7f7e0c0389e0 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.447+0000 7f7dfb7fe640 10 monclient(hunting): dump: 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:07.446 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4116055877 0 0) 0x7f7e0c0c5650 con 0x7f7e0c0c2ab0 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e00004eb0 con 0x7f7e0c0c2ab0 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3636379577 0 0) 0x7f7e0c009ea0 con 0x7f7e0c0389e0 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 >> v1:172.21.15.136:6789/0 conn(0x7f7e0c030660 legacy=0x7f7e0c0c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 >> v1:172.21.15.64:6789/0 conn(0x7f7e0c0c2ab0 legacy=0x7f7e0c16c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 monclient: found mon.c 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:07.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c00d290 con 0x7f7e0c0389e0 2024-03-29T11:33:07.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:07.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:07.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.448+0000 7f7dfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:07.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e1404f730 con 0x7f7e0c0389e0 2024-03-29T11:33:07.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7dfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:07.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7e14067d90 con 0x7f7e0c0389e0 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7dfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7dfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7dfb7fe640 10 monclient: dump: 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:07.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 10 monclient: _renew_subs 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c00d540 con 0x7f7e0c0389e0 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 10 monclient: _renew_subs 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c0c7e30 con 0x7f7e0c0389e0 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.449+0000 7f7e13fff640 1 librados: init done 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.450+0000 7f7e117fa640 4 set_mon_vals no callback set 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.451+0000 7f7e13fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.452+0000 7f7e13fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.452+0000 7f7e13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:07.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.452+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df4005080 con 0x7f7e0c0389e0 2024-03-29T11:33:07.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.453+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f7e14067ef0 con 0x7f7e0c0389e0 2024-03-29T11:33:07.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.455+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f7e140ff210 con 0x7f7e0c0389e0 2024-03-29T11:33:07.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.455+0000 7f7dfb7fe640 10 client.4574.objecter ms_dispatch 0x7f7e14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:07.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.455+0000 7f7dfb7fe640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:07.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.455+0000 7f7dfb7fe640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:07.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.455+0000 7f7dfb7fe640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:33:07.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.456+0000 7f7e12ffd640 10 monclient: get_auth_request con 0x7f7e0006dce0 auth_method 0 2024-03-29T11:33:07.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.457+0000 7f7dfb7fe640 10 client.4574.objecter ms_handle_connect 0x7f7e0006dce0 2024-03-29T11:33:07.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.464+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e140caaf0 con 0x7f7e0c0389e0 2024-03-29T11:33:07.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.466+0000 7f7dfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:07.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.466+0000 7f7dfb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:07.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7e13fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:07.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7e13fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:07.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7e13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:07.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7df4005330 con 0x7f7e0c0389e0 2024-03-29T11:33:07.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7dfb7fe640 1 -- 172.21.15.64:0/4178881184 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f7e140cca20 con 0x7f7e0c0389e0 2024-03-29T11:33:07.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7dfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:07.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.793+0000 7f7dfb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:07.792 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:07.792 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:07.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.800+0000 7f7e13fff640 10 librados: watch_flush enter 2024-03-29T11:33:07.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.800+0000 7f7e13fff640 10 librados: watch_flush exit 2024-03-29T11:33:07.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.800+0000 7f7e13fff640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:33:07.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.800+0000 7f7e13fff640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:33:07.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 >> v1:172.21.15.136:6816/2312274743 conn(0x7f7e0006dce0 legacy=0x7f7e000701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 10 monclient: shutdown 2024-03-29T11:33:07.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:07.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 >> v1:172.21.15.64:6790/0 conn(0x7f7e0c0389e0 legacy=0x7f7e0c0c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 shutdown_connections 2024-03-29T11:33:07.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 >> 172.21.15.64:0/4178881184 conn(0x7f7e0c005c00 msgr2=0x7f7e0c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 shutdown_connections 2024-03-29T11:33:07.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.801+0000 7f7e13fff640 1 -- 172.21.15.64:0/4178881184 wait complete. 2024-03-29T11:33:07.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:07.802+0000 7f7e13fff640 1 librados: shutdown 2024-03-29T11:33:07.816 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:14.623 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:14.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.773+0000 7fc0fac96640 10 monclient: get_monmap_and_config 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.773+0000 7fc0fac96640 10 monclient: build_initial_monmap 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.773+0000 7fc0fac96640 10 monclient: monmap: 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:14.774766+0000 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:14.774766+0000 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:14.770 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:14.771 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.774+0000 7fc0fac96640 1 Processor -- start 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.775+0000 7fc0fac96640 1 -- start start 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.775+0000 7fc0fac96640 10 monclient: init 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.775+0000 7fc0fac96640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.775+0000 7fc0fac96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.775+0000 7fc0fac96640 10 monclient(hunting): picked mon.noname-b con 0x7fc0ec0389e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:14.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.775+0000 7fc0fac96640 10 monclient(hunting): picked mon.noname-c con 0x7fc0ec038dc0 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:14.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.778+0000 7fc0fac96640 10 monclient(hunting): picked mon.noname-a con 0x7fc0ec030660 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:14.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.778+0000 7fc0fac96640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec0c54c0 con 0x7fc0ec030660 2024-03-29T11:33:14.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.778+0000 7fc0fac96640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec036430 con 0x7fc0ec038dc0 2024-03-29T11:33:14.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.778+0000 7fc0fac96640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec039840 con 0x7fc0ec0389e0 2024-03-29T11:33:14.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.778+0000 7fc0fac96640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:14.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.778+0000 7fc0fac96640 10 monclient(hunting): authenticate will time out at 1731.048759s 2024-03-29T11:33:14.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f9233640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fc0ec030660 0x7fc0ec0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58433/0 (socket says 172.21.15.64:58433) 2024-03-29T11:33:14.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f9233640 1 -- 172.21.15.64:0/2974999817 learned_addr learned my addr 172.21.15.64:0/2974999817 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345207452 0 0) 0x7fc0ec0c54c0 con 0x7fc0ec030660 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.781+0000 7fc0f37fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.782+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e4003520 con 0x7fc0ec030660 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.782+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2118904211 0 0) 0x7fc0e4003520 con 0x7fc0ec030660 2024-03-29T11:33:14.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.782+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 >> v1:172.21.15.64:6790/0 conn(0x7fc0ec0389e0 legacy=0x7fc0ec033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:14.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 >> v1:172.21.15.136:6789/0 conn(0x7fc0ec038dc0 legacy=0x7fc0ec0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:14.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:14.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 1 monclient: found mon.noname-a 2024-03-29T11:33:14.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec035690 con 0x7fc0ec030660 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc0f404f730 con 0x7fc0ec030660 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.783+0000 7fc0f37fe640 10 monclient: dump: 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.783 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0f4075f10 con 0x7fc0ec030660 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0f37fe640 1 -- 172.21.15.64:0/2974999817 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc0f40758b0 con 0x7fc0ec030660 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0f37fe640 10 monclient: dump: 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.784 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0fac96640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0fac96640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.785+0000 7fc0fac96640 4 set_mon_vals no callback set 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.786+0000 7fc0fac96640 10 monclient: shutdown 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.786+0000 7fc0fac96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.786+0000 7fc0fac96640 1 -- 172.21.15.64:0/2974999817 >> v1:172.21.15.64:6789/0 conn(0x7fc0ec030660 legacy=0x7fc0ec0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.786+0000 7fc0fac96640 1 -- 172.21.15.64:0/2974999817 shutdown_connections 2024-03-29T11:33:14.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.786+0000 7fc0fac96640 1 -- 172.21.15.64:0/2974999817 >> 172.21.15.64:0/2974999817 conn(0x7fc0ec005c00 msgr2=0x7fc0ec006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.787+0000 7fc0fac96640 1 -- 172.21.15.64:0/2974999817 shutdown_connections 2024-03-29T11:33:14.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.787+0000 7fc0fac96640 1 -- 172.21.15.64:0/2974999817 wait complete. 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.788+0000 7fc0fac96640 10 monclient: build_initial_monmap 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.788+0000 7fc0fac96640 10 monclient: monmap: 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.788+0000 7fc0fac96640 1 librados: starting msgr at 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.788+0000 7fc0fac96640 1 librados: starting objecter 2024-03-29T11:33:14.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.789+0000 7fc0fac96640 1 Processor -- start 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.789+0000 7fc0fac96640 1 -- start start 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.789+0000 7fc0fac96640 1 librados: setting wanted keys 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.789+0000 7fc0fac96640 1 librados: calling monclient init 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.789+0000 7fc0fac96640 10 monclient: init 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.790+0000 7fc0fac96640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.790+0000 7fc0fac96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.790+0000 7fc0fac96640 10 monclient(hunting): picked mon.noname-b con 0x7fc0ec030660 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:14.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.790+0000 7fc0fac96640 10 monclient(hunting): picked mon.noname-c con 0x7fc0ec0389e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:14.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.791+0000 7fc0fac96640 10 monclient(hunting): picked mon.noname-a con 0x7fc0ec169e80 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:14.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.791+0000 7fc0fac96640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec02ffe0 con 0x7fc0ec169e80 2024-03-29T11:33:14.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.791+0000 7fc0fac96640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec15fbc0 con 0x7fc0ec030660 2024-03-29T11:33:14.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.791+0000 7fc0fac96640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec02d150 con 0x7fc0ec0389e0 2024-03-29T11:33:14.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.791+0000 7fc0fac96640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:14.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.791+0000 7fc0fac96640 10 monclient(hunting): authenticate will time out at 1731.061647s 2024-03-29T11:33:14.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.793+0000 7fc0f9233640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fc0ec169e80 0x7fc0ec15f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58547/0 (socket says 172.21.15.64:58547) 2024-03-29T11:33:14.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f9233640 1 -- 172.21.15.64:0/2024629011 learned_addr learned my addr 172.21.15.64:0/2024629011 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 10 client.?.objecter ms_handle_connect 0x7fc0ec169e80 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212592572 0 0) 0x7fc0ec02ffe0 con 0x7fc0ec169e80 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.794+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d8004eb0 con 0x7fc0ec169e80 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2267243712 0 0) 0x7fc0d8004eb0 con 0x7fc0ec169e80 2024-03-29T11:33:14.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0ec02ffe0 con 0x7fc0ec169e80 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc0f404f400 con 0x7fc0ec169e80 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 10 monclient(hunting): dump: 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:14.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.795+0000 7fc0f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:14.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.796+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4042758547 0 0) 0x7fc0ec02ffe0 con 0x7fc0ec169e80 2024-03-29T11:33:14.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.796+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 >> v1:172.21.15.64:6790/0 conn(0x7fc0ec0389e0 legacy=0x7fc0ec15eda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:14.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 >> v1:172.21.15.136:6789/0 conn(0x7fc0ec030660 legacy=0x7fc0ec169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:33:14.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 1 monclient: found mon.a 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec02d2e0 con 0x7fc0ec169e80 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 monclient: _finish_auth 0 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 client.?.objecter ms_handle_connect 0x7fc0ec030660 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0fac96640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0fac96640 10 monclient: _renew_subs 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0fac96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec0c83d0 con 0x7fc0ec169e80 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0f404f730 con 0x7fc0ec169e80 2024-03-29T11:33:14.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.797+0000 7fc0f2ffd640 4 set_mon_vals no callback set 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0fac96640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0fac96640 10 monclient: _renew_subs 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0fac96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec02eb30 con 0x7fc0ec169e80 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0fac96640 1 librados: init done 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fc0f404f400 con 0x7fc0ec169e80 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.798+0000 7fc0f17fa640 10 monclient: dump: 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.796 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.799+0000 7fc0fac96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.799+0000 7fc0fac96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.799+0000 7fc0fac96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:33:14.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.799+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ec007b70 con 0x7fc0ec169e80 2024-03-29T11:33:14.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.803+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fc0f4067bf0 con 0x7fc0ec169e80 2024-03-29T11:33:14.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.804+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fc0f40ff490 con 0x7fc0ec169e80 2024-03-29T11:33:14.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.804+0000 7fc0f17fa640 10 client.4542.objecter ms_dispatch 0x7fc0f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:14.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.804+0000 7fc0f17fa640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:14.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.804+0000 7fc0f17fa640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:14.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.804+0000 7fc0f17fa640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:33:14.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.805+0000 7fc0f8a32640 10 monclient: get_auth_request con 0x7fc0d806dc50 auth_method 0 2024-03-29T11:33:14.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.805+0000 7fc0f17fa640 10 client.4542.objecter ms_handle_connect 0x7fc0d806dc50 2024-03-29T11:33:14.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.812+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0f40cad70 con 0x7fc0ec169e80 2024-03-29T11:33:14.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.812+0000 7fc0f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:14.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:14.812+0000 7fc0f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:15.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.144+0000 7fc0fac96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:15.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.144+0000 7fc0fac96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:15.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.144+0000 7fc0fac96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:33:15.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.144+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0ec005120 con 0x7fc0ec169e80 2024-03-29T11:33:15.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.146+0000 7fc0f17fa640 1 -- 172.21.15.64:0/2024629011 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fc0f40ccca0 con 0x7fc0ec169e80 2024-03-29T11:33:15.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.146+0000 7fc0f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:15.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.146+0000 7fc0f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:15.142 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:15.143 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:15.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.150+0000 7fc0fac96640 10 librados: watch_flush enter 2024-03-29T11:33:15.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.150+0000 7fc0fac96640 10 librados: watch_flush exit 2024-03-29T11:33:15.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.150+0000 7fc0fac96640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:33:15.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.150+0000 7fc0fac96640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:33:15.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.151+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 >> v1:172.21.15.136:6816/2312274743 conn(0x7fc0d806dc50 legacy=0x7fc0d8070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.151+0000 7fc0fac96640 10 monclient: shutdown 2024-03-29T11:33:15.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.151+0000 7fc0fac96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:15.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.151+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 >> v1:172.21.15.64:6789/0 conn(0x7fc0ec169e80 legacy=0x7fc0ec15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:15.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.152+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 shutdown_connections 2024-03-29T11:33:15.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.152+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 >> 172.21.15.64:0/2024629011 conn(0x7fc0ec005c00 msgr2=0x7fc0ec03ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:15.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.153+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 shutdown_connections 2024-03-29T11:33:15.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.153+0000 7fc0fac96640 1 -- 172.21.15.64:0/2024629011 wait complete. 2024-03-29T11:33:15.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:15.153+0000 7fc0fac96640 1 librados: shutdown 2024-03-29T11:33:15.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:21.977 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:22.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.136+0000 7f3b94c2d640 10 monclient: get_monmap_and_config 2024-03-29T11:33:22.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.136+0000 7f3b94c2d640 10 monclient: build_initial_monmap 2024-03-29T11:33:22.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.136+0000 7f3b94c2d640 10 monclient: monmap: 2024-03-29T11:33:22.132 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:22.137082+0000 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:22.137082+0000 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:22.133 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.137+0000 7f3b94c2d640 1 Processor -- start 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.137+0000 7f3b94c2d640 1 -- start start 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.137+0000 7f3b94c2d640 10 monclient: init 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b900b2790 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b90143c80 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b90146e30 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b90146290 con 0x7f3b900b2790 2024-03-29T11:33:22.135 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b900b3970 con 0x7f3b90146e30 2024-03-29T11:33:22.136 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9006ee80 con 0x7f3b90143c80 2024-03-29T11:33:22.136 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:22.136 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.138+0000 7f3b94c2d640 10 monclient(hunting): authenticate will time out at 1738.408180s 2024-03-29T11:33:22.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.141+0000 7f3b8effd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3b90143c80 0x7f3b90146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40937/0 (socket says 172.21.15.64:40937) 2024-03-29T11:33:22.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.141+0000 7f3b8effd640 1 -- 172.21.15.64:0/3484544231 learned_addr learned my addr 172.21.15.64:0/3484544231 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:22.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660807928 0 0) 0x7f3b9006ee80 con 0x7f3b90143c80 2024-03-29T11:33:22.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b78003520 con 0x7f3b90143c80 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757386296 0 0) 0x7f3b900b3970 con 0x7f3b90146e30 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9006ee80 con 0x7f3b90146e30 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.142+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1531627823 0 0) 0x7f3b78003520 con 0x7f3b90143c80 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.143+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 >> v1:172.21.15.136:6789/0 conn(0x7f3b90146e30 legacy=0x7f3b90151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.143+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 >> v1:172.21.15.64:6789/0 conn(0x7f3b900b2790 legacy=0x7f3b90143570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:22.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9006f160 con 0x7f3b90143c80 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b94c2d640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b94c2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b88001a10 con 0x7f3b90143c80 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:22.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: dump: 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b94c2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b88001c30 con 0x7f3b90143c80 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:22.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b94c2d640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.144+0000 7f3b94c2d640 4 set_mon_vals no callback set 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b8e7fc640 1 -- 172.21.15.64:0/3484544231 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b88001fc0 con 0x7f3b90143c80 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b8e7fc640 10 monclient: dump: 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:22.144 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b94c2d640 10 monclient: shutdown 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b94c2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b94c2d640 1 -- 172.21.15.64:0/3484544231 >> v1:172.21.15.64:6790/0 conn(0x7f3b90143c80 legacy=0x7f3b90146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b94c2d640 1 -- 172.21.15.64:0/3484544231 shutdown_connections 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.145+0000 7f3b94c2d640 1 -- 172.21.15.64:0/3484544231 >> 172.21.15.64:0/3484544231 conn(0x7f3b9006d070 msgr2=0x7f3b90066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.148+0000 7f3b94c2d640 1 -- 172.21.15.64:0/3484544231 shutdown_connections 2024-03-29T11:33:22.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.149+0000 7f3b94c2d640 1 -- 172.21.15.64:0/3484544231 wait complete. 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.149+0000 7f3b94c2d640 10 monclient: build_initial_monmap 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.149+0000 7f3b94c2d640 10 monclient: monmap: 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:22.147 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 1 librados: starting msgr at 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 1 librados: starting objecter 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 1 Processor -- start 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 1 -- start start 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 1 librados: setting wanted keys 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 1 librados: calling monclient init 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.150+0000 7f3b94c2d640 10 monclient: init 2024-03-29T11:33:22.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.151+0000 7f3b94c2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:22.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.151+0000 7f3b94c2d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:22.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.151+0000 7f3b94c2d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b900b2790 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:22.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.155+0000 7f3b94c2d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b90143c80 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b8effd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3b90143c80 0x7f3b901405b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41025/0 (socket says 172.21.15.64:41025) 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b94c2d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b90146e30 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b8effd640 1 -- 172.21.15.64:0/2219466137 learned_addr learned my addr 172.21.15.64:0/2219466137 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b901dc240 con 0x7f3b900b2790 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9013fa60 con 0x7f3b90146e30 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b90140090 con 0x7f3b90143c80 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b94c2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.158+0000 7f3b94c2d640 10 monclient(hunting): authenticate will time out at 1738.428815s 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 client.?.objecter ms_handle_connect 0x7f3b90143c80 2024-03-29T11:33:22.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:22.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 client.?.objecter ms_handle_connect 0x7f3b90146e30 2024-03-29T11:33:22.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:22.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 client.?.objecter ms_handle_connect 0x7f3b900b2790 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176997124 0 0) 0x7f3b90140090 con 0x7f3b90143c80 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b7c004eb0 con 0x7f3b90143c80 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165318287 0 0) 0x7f3b9013fa60 con 0x7f3b90146e30 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b90140090 con 0x7f3b90146e30 2024-03-29T11:33:22.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636500383 0 0) 0x7f3b901dc240 con 0x7f3b900b2790 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.159+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9013fa60 con 0x7f3b900b2790 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2524855845 0 0) 0x7f3b90140090 con 0x7f3b90146e30 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b7c006680 con 0x7f3b90146e30 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b84001850 con 0x7f3b90146e30 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 10 monclient(hunting): dump: 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:22.162 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.160+0000 7f3b77fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872122397 0 0) 0x7f3b7c004eb0 con 0x7f3b90143c80 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b90140090 con 0x7f3b90143c80 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b88001f40 con 0x7f3b90143c80 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:22.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient(hunting): dump: 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254914117 0 0) 0x7f3b90140090 con 0x7f3b90143c80 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 >> v1:172.21.15.136:6789/0 conn(0x7f3b90146e30 legacy=0x7f3b901d86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 >> v1:172.21.15.64:6789/0 conn(0x7f3b900b2790 legacy=0x7f3b901db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 monclient: found mon.c 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b901d8df0 con 0x7f3b90143c80 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient: _finish_auth 0 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b94c2d640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b94c2d640 10 monclient: _renew_subs 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b94c2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.161+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9013e800 con 0x7f3b90143c80 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.162+0000 7f3b94c2d640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.162+0000 7f3b94c2d640 10 monclient: _renew_subs 2024-03-29T11:33:22.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.162+0000 7f3b94c2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.162+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b900b6a60 con 0x7f3b90143c80 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.162+0000 7f3b94c2d640 1 librados: init done 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.163+0000 7f3b94c2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.163+0000 7f3b94c2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.163+0000 7f3b94c2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.163+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b90140220 con 0x7f3b90143c80 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b880020e0 con 0x7f3b90143c80 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f3b880015b0 con 0x7f3b90143c80 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b77fff640 10 monclient: dump: 2024-03-29T11:33:22.166 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:22.167 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:22.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.164+0000 7f3b8dffb640 4 set_mon_vals no callback set 2024-03-29T11:33:22.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.165+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f3b8801b040 con 0x7f3b90143c80 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.165+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f3b88054cd0 con 0x7f3b90143c80 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.165+0000 7f3b77fff640 10 client.4580.objecter ms_dispatch 0x7f3b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.165+0000 7f3b77fff640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.165+0000 7f3b77fff640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.165+0000 7f3b77fff640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.166+0000 7f3b8f7fe640 10 monclient: get_auth_request con 0x7f3b7c06e1e0 auth_method 0 2024-03-29T11:33:22.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.167+0000 7f3b77fff640 10 client.4580.objecter ms_handle_connect 0x7f3b7c06e1e0 2024-03-29T11:33:22.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.174+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b88050f70 con 0x7f3b90143c80 2024-03-29T11:33:22.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.174+0000 7f3b77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:22.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.174+0000 7f3b77fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:22.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.515+0000 7f3b94c2d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:22.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.515+0000 7f3b94c2d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:22.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.515+0000 7f3b94c2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:22.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.515+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b901d8f80 con 0x7f3b90143c80 2024-03-29T11:33:22.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.516+0000 7f3b77fff640 1 -- 172.21.15.64:0/2219466137 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f3b88050f70 con 0x7f3b90143c80 2024-03-29T11:33:22.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.516+0000 7f3b77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:22.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.516+0000 7f3b77fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:22.512 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:22.513 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 10 librados: watch_flush enter 2024-03-29T11:33:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 10 librados: watch_flush exit 2024-03-29T11:33:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:33:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 >> v1:172.21.15.136:6816/2312274743 conn(0x7f3b7c06e1e0 legacy=0x7f3b7c0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 10 monclient: shutdown 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 >> v1:172.21.15.64:6790/0 conn(0x7f3b90143c80 legacy=0x7f3b901405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 shutdown_connections 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 >> 172.21.15.64:0/2219466137 conn(0x7f3b9006d070 msgr2=0x7f3b900b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 shutdown_connections 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.520+0000 7f3b94c2d640 1 -- 172.21.15.64:0/2219466137 wait complete. 2024-03-29T11:33:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:22.521+0000 7f3b94c2d640 1 librados: shutdown 2024-03-29T11:33:22.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:29.342 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:29.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.494+0000 7f7693af1640 10 monclient: get_monmap_and_config 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.494+0000 7f7693af1640 10 monclient: build_initial_monmap 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.494+0000 7f7693af1640 10 monclient: monmap: 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:29.495468+0000 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:29.495468+0000 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:29.491 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:29.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.495+0000 7f7693af1640 1 Processor -- start 2024-03-29T11:33:29.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.495+0000 7f7693af1640 1 -- start start 2024-03-29T11:33:29.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.495+0000 7f7693af1640 10 monclient: init 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient(hunting): picked mon.noname-a con 0x7f768c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient(hunting): picked mon.noname-c con 0x7f768c13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient(hunting): picked mon.noname-b con 0x7f768c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0aaef0 con 0x7f768c1457f0 2024-03-29T11:33:29.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c145bd0 con 0x7f768c13f650 2024-03-29T11:33:29.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c145eb0 con 0x7f768c0a9d70 2024-03-29T11:33:29.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:29.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.496+0000 7f7693af1640 10 monclient(hunting): authenticate will time out at 1745.766692s 2024-03-29T11:33:29.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.499+0000 7f769108c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f768c13f650 0x7f768c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:49823/0 (socket says 172.21.15.64:49823) 2024-03-29T11:33:29.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.499+0000 7f769108c640 1 -- 172.21.15.64:0/2225180682 learned_addr learned my addr 172.21.15.64:0/2225180682 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299650430 0 0) 0x7f768c145bd0 con 0x7f768c13f650 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): _negotiate 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7678003520 con 0x7f768c13f650 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47561899 0 0) 0x7f768c145eb0 con 0x7f768c0a9d70 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): _negotiate 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:33:29.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.502+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c145bd0 con 0x7f768c0a9d70 2024-03-29T11:33:29.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002128505 0 0) 0x7f7678003520 con 0x7f768c13f650 2024-03-29T11:33:29.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 >> v1:172.21.15.64:6790/0 conn(0x7f768c0a9d70 legacy=0x7f768c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 >> v1:172.21.15.64:6789/0 conn(0x7f768c1457f0 legacy=0x7f768c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 10 monclient: _finish_hunting 0 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 1 monclient: found mon.noname-c 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.504+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c146340 con 0x7f768c13f650 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: _finish_auth 0 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: _check_auth_tickets 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7684001a10 con 0x7f768c13f650 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: dump: 2024-03-29T11:33:29.503 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f7693af1640 5 monclient: authenticate success, global_id 4588 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f7693af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7684001c30 con 0x7f768c13f650 2024-03-29T11:33:29.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 1 -- 172.21.15.64:0/2225180682 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7684001fc0 con 0x7f768c13f650 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.505+0000 7f769088b640 10 monclient: dump: 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:29.505 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:29.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.507+0000 7f7693af1640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:29.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.507+0000 7f7693af1640 4 set_mon_vals no callback set 2024-03-29T11:33:29.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.509+0000 7f7693af1640 10 monclient: shutdown 2024-03-29T11:33:29.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.509+0000 7f7693af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:29.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.509+0000 7f7693af1640 1 -- 172.21.15.64:0/2225180682 >> v1:172.21.15.136:6789/0 conn(0x7f768c13f650 legacy=0x7f768c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.511+0000 7f7693af1640 1 -- 172.21.15.64:0/2225180682 shutdown_connections 2024-03-29T11:33:29.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.511+0000 7f7693af1640 1 -- 172.21.15.64:0/2225180682 >> 172.21.15.64:0/2225180682 conn(0x7f768c066a00 msgr2=0x7f768c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:29.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.512+0000 7f7693af1640 1 -- 172.21.15.64:0/2225180682 shutdown_connections 2024-03-29T11:33:29.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.513+0000 7f7693af1640 1 -- 172.21.15.64:0/2225180682 wait complete. 2024-03-29T11:33:29.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.513+0000 7f7693af1640 10 monclient: build_initial_monmap 2024-03-29T11:33:29.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.513+0000 7f7693af1640 10 monclient: monmap: 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:29.512 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 1 librados: starting msgr at 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 1 librados: starting objecter 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 1 Processor -- start 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 1 -- start start 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 1 librados: setting wanted keys 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 1 librados: calling monclient init 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.514+0000 7f7693af1640 10 monclient: init 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient(hunting): picked mon.noname-b con 0x7f768c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient(hunting): picked mon.noname-c con 0x7f768c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient(hunting): picked mon.noname-a con 0x7f768c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c13c8e0 con 0x7f768c1457f0 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c136690 con 0x7f768c0a9d70 2024-03-29T11:33:29.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c1358a0 con 0x7f768c13f650 2024-03-29T11:33:29.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:29.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.515+0000 7f7693af1640 10 monclient(hunting): authenticate will time out at 1745.785158s 2024-03-29T11:33:29.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.520+0000 7f769108c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f768c13f650 0x7f768c071740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:60327/0 (socket says 172.21.15.64:60327) 2024-03-29T11:33:29.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.520+0000 7f769108c640 1 -- 172.21.15.64:0/3502850439 learned_addr learned my addr 172.21.15.64:0/3502850439 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:29.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.520+0000 7f76827fc640 10 client.?.objecter ms_handle_connect 0x7f768c13f650 2024-03-29T11:33:29.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.520+0000 7f76827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 client.?.objecter ms_handle_connect 0x7f768c0a9d70 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428642161 0 0) 0x7f768c1358a0 con 0x7f768c13f650 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7670004eb0 con 0x7f768c13f650 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589046479 0 0) 0x7f7670004eb0 con 0x7f768c13f650 2024-03-29T11:33:29.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f768c1358a0 con 0x7f768c13f650 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f7684001a90 con 0x7f768c13f650 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): dump: 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:29.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.522+0000 7f76827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2021827206 0 0) 0x7f768c1358a0 con 0x7f768c13f650 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 >> v1:172.21.15.136:6789/0 conn(0x7f768c0a9d70 legacy=0x7f768c1424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 >> v1:172.21.15.64:6789/0 conn(0x7f768c1457f0 legacy=0x7f768c1ffdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 1 monclient: found mon.c 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c071e50 con 0x7f768c13f650 2024-03-29T11:33:29.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f76827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 10 monclient: _renew_subs 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c06ff80 con 0x7f768c13f650 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 10 monclient: _renew_subs 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c070c60 con 0x7f768c13f650 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.523+0000 7f7693af1640 1 librados: init done 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f7693af1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7684004e10 con 0x7f768c13f650 2024-03-29T11:33:29.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f76827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f76840051e0 con 0x7f768c13f650 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f76827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f76827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f76827fc640 10 monclient: dump: 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:29.524 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:29.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.525+0000 7f7683fff640 4 set_mon_vals no callback set 2024-03-29T11:33:29.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f7684005400 con 0x7f768c13f650 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f7684085080 con 0x7f768c13f650 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f76827fc640 10 client.4586.objecter ms_dispatch 0x7f768c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f76827fc640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f76827fc640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f76827fc640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.526+0000 7f7693af1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.527+0000 7f7693af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.527+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c075f10 con 0x7f768c13f650 2024-03-29T11:33:29.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.529+0000 7f769188d640 10 monclient: get_auth_request con 0x7f767006dc60 auth_method 0 2024-03-29T11:33:29.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.530+0000 7f76827fc640 10 client.4586.objecter ms_handle_connect 0x7f767006dc60 2024-03-29T11:33:29.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.540+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7684050960 con 0x7f768c13f650 2024-03-29T11:33:29.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.540+0000 7f76827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:29.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.540+0000 7f76827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:29.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.870+0000 7f7693af1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:29.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.870+0000 7f7693af1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:29.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.870+0000 7f7693af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:29.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.870+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f768c0b3680 con 0x7f768c13f650 2024-03-29T11:33:29.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.871+0000 7f76827fc640 1 -- 172.21.15.64:0/3502850439 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f7684052890 con 0x7f768c13f650 2024-03-29T11:33:29.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.871+0000 7f76827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:29.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.871+0000 7f76827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:29.868 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:29.868 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:29.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.874+0000 7f7693af1640 10 librados: watch_flush enter 2024-03-29T11:33:29.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.874+0000 7f7693af1640 10 librados: watch_flush exit 2024-03-29T11:33:29.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.874+0000 7f7693af1640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:33:29.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.874+0000 7f7693af1640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:33:29.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.879+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 >> v1:172.21.15.136:6816/2312274743 conn(0x7f767006dc60 legacy=0x7f7670070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.879+0000 7f7693af1640 10 monclient: shutdown 2024-03-29T11:33:29.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.879+0000 7f7693af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:29.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.879+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 >> v1:172.21.15.64:6790/0 conn(0x7f768c13f650 legacy=0x7f768c071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.880+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 shutdown_connections 2024-03-29T11:33:29.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.880+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 >> 172.21.15.64:0/3502850439 conn(0x7f768c066a00 msgr2=0x7f768c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:29.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.880+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 shutdown_connections 2024-03-29T11:33:29.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.880+0000 7f7693af1640 1 -- 172.21.15.64:0/3502850439 wait complete. 2024-03-29T11:33:29.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:29.880+0000 7f7693af1640 1 librados: shutdown 2024-03-29T11:33:29.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:36.701 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:36.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.867+0000 7f0b71a1b640 10 monclient: get_monmap_and_config 2024-03-29T11:33:36.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.867+0000 7f0b71a1b640 10 monclient: build_initial_monmap 2024-03-29T11:33:36.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.867+0000 7f0b71a1b640 10 monclient: monmap: 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:36.868193+0000 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:36.868193+0000 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:36.864 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:36.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.868+0000 7f0b71a1b640 1 Processor -- start 2024-03-29T11:33:36.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.868+0000 7f0b71a1b640 1 -- start start 2024-03-29T11:33:36.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.868+0000 7f0b71a1b640 10 monclient: init 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.869+0000 7f0b71a1b640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.869+0000 7f0b71a1b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.869+0000 7f0b71a1b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b640389e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.869+0000 7f0b71a1b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b64038dc0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b70a19640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0b640389e0 0x7f0b64033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:35802/0 (socket says 172.21.15.64:35802) 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b71a1b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b64030660 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:36.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b70a19640 1 -- 172.21.15.64:0/2985171044 learned_addr learned my addr 172.21.15.64:0/2985171044 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:36.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640c54c0 con 0x7f0b64030660 2024-03-29T11:33:36.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b64036430 con 0x7f0b640389e0 2024-03-29T11:33:36.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b64039840 con 0x7f0b64038dc0 2024-03-29T11:33:36.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b71a1b640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:36.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.870+0000 7f0b71a1b640 10 monclient(hunting): authenticate will time out at 1753.140726s 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291062799 0 0) 0x7f0b64036430 con 0x7f0b640389e0 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b5c003520 con 0x7f0b640389e0 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.871+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 112382714 0 0) 0x7f0b5c003520 con 0x7f0b640389e0 2024-03-29T11:33:36.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.872+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 >> v1:172.21.15.64:6790/0 conn(0x7f0b64038dc0 legacy=0x7f0b640cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 >> v1:172.21.15.64:6789/0 conn(0x7f0b64030660 legacy=0x7f0b640d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64035690 con 0x7f0b640389e0 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0b60001690 con 0x7f0b640389e0 2024-03-29T11:33:36.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: dump: 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b600018f0 con 0x7f0b640389e0 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:36.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 1 -- 172.21.15.64:0/2985171044 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0b60001d40 con 0x7f0b640389e0 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b6b7fe640 10 monclient: dump: 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b71a1b640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b71a1b640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:36.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.874+0000 7f0b71a1b640 4 set_mon_vals no callback set 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.875+0000 7f0b71a1b640 10 monclient: shutdown 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.875+0000 7f0b71a1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.875+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 >> v1:172.21.15.136:6789/0 conn(0x7f0b640389e0 legacy=0x7f0b64033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.876+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 shutdown_connections 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.876+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 >> 172.21.15.64:0/2985171044 conn(0x7f0b64005c00 msgr2=0x7f0b64006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.876+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 shutdown_connections 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.876+0000 7f0b71a1b640 1 -- 172.21.15.64:0/2985171044 wait complete. 2024-03-29T11:33:36.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.877+0000 7f0b71a1b640 10 monclient: build_initial_monmap 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.877+0000 7f0b71a1b640 10 monclient: monmap: 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 1 librados: starting msgr at 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 1 librados: starting objecter 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 1 Processor -- start 2024-03-29T11:33:36.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 1 -- start start 2024-03-29T11:33:36.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 1 librados: setting wanted keys 2024-03-29T11:33:36.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 1 librados: calling monclient init 2024-03-29T11:33:36.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.878+0000 7f0b71a1b640 10 monclient: init 2024-03-29T11:33:36.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.879+0000 7f0b71a1b640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:36.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.879+0000 7f0b71a1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.879+0000 7f0b71a1b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b64030660 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.879+0000 7f0b71a1b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b640389e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b71a1b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6402d860 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b71a1b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640bf560 con 0x7f0b64030660 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b71a1b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6402e760 con 0x7f0b6402d860 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b71a1b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6402f4a0 con 0x7f0b640389e0 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b71a1b640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b71a1b640 10 monclient(hunting): authenticate will time out at 1753.149921s 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b7121a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0b6402d860 0x7f0b640bee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:46178/0 (socket says 172.21.15.64:46178) 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b7121a640 1 -- 172.21.15.64:0/4145565145 learned_addr learned my addr 172.21.15.64:0/4145565145 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 10 client.?.objecter ms_handle_connect 0x7f0b6402d860 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187657075 0 0) 0x7f0b6402e760 con 0x7f0b6402d860 2024-03-29T11:33:36.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:36.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:36.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:36.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:33:36.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.880+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b50004eb0 con 0x7f0b6402d860 2024-03-29T11:33:36.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350571877 0 0) 0x7f0b50004eb0 con 0x7f0b6402d860 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b6402e760 con 0x7f0b6402d860 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0b6c04fdd0 con 0x7f0b6402d860 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 10 monclient(hunting): dump: 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:36.881 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.881+0000 7f0b697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.883+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882717962 0 0) 0x7f0b6402e760 con 0x7f0b6402d860 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.883+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 >> v1:172.21.15.64:6790/0 conn(0x7f0b640389e0 legacy=0x7f0b640be740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.883+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 >> v1:172.21.15.64:6789/0 conn(0x7f0b64030660 legacy=0x7f0b6402d150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 1 monclient: found mon.b 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6402f630 con 0x7f0b6402d860 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: _finish_auth 0 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 5 monclient: authenticate success, global_id 4594 2024-03-29T11:33:36.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 10 monclient: _renew_subs 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b640c9e90 con 0x7f0b6402d860 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 10 monclient: _renew_subs 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6402ed00 con 0x7f0b6402d860 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b71a1b640 1 librados: init done 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b6c04f400 con 0x7f0b6402d860 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f0b6c075710 con 0x7f0b6402d860 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.884+0000 7f0b697fa640 10 monclient: dump: 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:36.883 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:36.884 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.885+0000 7f0b71a1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.885+0000 7f0b6affd640 4 set_mon_vals no callback set 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.886+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f0b6c067bf0 con 0x7f0b6402d860 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f0b6c0ff600 con 0x7f0b6402d860 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b697fa640 10 client.4594.objecter ms_dispatch 0x7f0b6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b697fa640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b697fa640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b697fa640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-29T11:33:36.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b71a1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:36.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b71a1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:36.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b64007b70 con 0x7f0b6402d860 2024-03-29T11:33:36.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b70a19640 10 monclient: get_auth_request con 0x7f0b5006db10 auth_method 0 2024-03-29T11:33:36.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.887+0000 7f0b697fa640 10 client.4594.objecter ms_handle_connect 0x7f0b5006db10 2024-03-29T11:33:36.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.894+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b6c0caf40 con 0x7f0b6402d860 2024-03-29T11:33:36.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.894+0000 7f0b697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:36.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:36.894+0000 7f0b697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:37.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.229+0000 7f0b71a1b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:37.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.229+0000 7f0b71a1b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:37.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.229+0000 7f0b71a1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:37.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.229+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b6402f7c0 con 0x7f0b6402d860 2024-03-29T11:33:37.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.229+0000 7f0b697fa640 1 -- 172.21.15.64:0/4145565145 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f0b6c0cce70 con 0x7f0b6402d860 2024-03-29T11:33:37.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.230+0000 7f0b697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:37.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.230+0000 7f0b697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:37.227 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:37.227 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.233+0000 7f0b71a1b640 10 librados: watch_flush enter 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.233+0000 7f0b71a1b640 10 librados: watch_flush exit 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.233+0000 7f0b71a1b640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.233+0000 7f0b71a1b640 10 client.4594.objecter successfully canceled tick 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.234+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 >> v1:172.21.15.136:6816/2312274743 conn(0x7f0b5006db10 legacy=0x7f0b5006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.234+0000 7f0b71a1b640 10 monclient: shutdown 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.234+0000 7f0b71a1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:37.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.234+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 >> v1:172.21.15.136:6789/0 conn(0x7f0b6402d860 legacy=0x7f0b640bee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:37.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.235+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 shutdown_connections 2024-03-29T11:33:37.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.235+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 >> 172.21.15.64:0/4145565145 conn(0x7f0b64005c00 msgr2=0x7f0b64005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:37.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.235+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 shutdown_connections 2024-03-29T11:33:37.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.235+0000 7f0b71a1b640 1 -- 172.21.15.64:0/4145565145 wait complete. 2024-03-29T11:33:37.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:37.236+0000 7f0b71a1b640 1 librados: shutdown 2024-03-29T11:33:37.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:44.056 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:44.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.213+0000 7f4bdc64e640 10 monclient: get_monmap_and_config 2024-03-29T11:33:44.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.213+0000 7f4bdc64e640 10 monclient: build_initial_monmap 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.213+0000 7f4bdc64e640 10 monclient: monmap: 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:44.214112+0000 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:44.214112+0000 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:44.210 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.214+0000 7f4bdc64e640 1 Processor -- start 2024-03-29T11:33:44.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.214+0000 7f4bdc64e640 1 -- start start 2024-03-29T11:33:44.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.214+0000 7f4bdc64e640 10 monclient: init 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient(hunting): picked mon.noname-b con 0x7f4bd41422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient(hunting): picked mon.noname-c con 0x7f4bd4142690 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient(hunting): picked mon.noname-a con 0x7f4bd40b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd4142c80 con 0x7f4bd40b17d0 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd40aae40 con 0x7f4bd4142690 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd413fbd0 con 0x7f4bd41422b0 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdc64e640 10 monclient(hunting): authenticate will time out at 1760.485167s 2024-03-29T11:33:44.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdabeb640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4bd40b17d0 0x7f4bd40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38636/0 (socket says 172.21.15.64:38636) 2024-03-29T11:33:44.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.215+0000 7f4bdabeb640 1 -- 172.21.15.64:0/385331619 learned_addr learned my addr 172.21.15.64:0/385331619 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244142909 0 0) 0x7f4bd4142c80 con 0x7f4bd40b17d0 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): _negotiate 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc0003520 con 0x7f4bd40b17d0 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077996729 0 0) 0x7f4bd40aae40 con 0x7f4bd4142690 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): _negotiate 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:44.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd4142c80 con 0x7f4bd4142690 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750894828 0 0) 0x7f4bc0003520 con 0x7f4bd40b17d0 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 >> v1:172.21.15.64:6790/0 conn(0x7f4bd41422b0 legacy=0x7f4bd413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 >> v1:172.21.15.136:6789/0 conn(0x7f4bd4142690 legacy=0x7f4bd40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: _finish_hunting 0 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 monclient: found mon.noname-a 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd4151940 con 0x7f4bd40b17d0 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: _finish_auth 0 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: _check_auth_tickets 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4bcc001a10 con 0x7f4bd40b17d0 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bd93e8640 10 monclient: dump: 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:44.216 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bdc64e640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:33:44.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.216+0000 7f4bdc64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:44.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bcc001b70 con 0x7f4bd40b17d0 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bd93e8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bd93e8640 1 -- 172.21.15.64:0/385331619 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4bcc001f00 con 0x7f4bd40b17d0 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bd93e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bd93e8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bd93e8640 10 monclient: dump: 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.225 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bdc64e640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.217+0000 7f4bdc64e640 4 set_mon_vals no callback set 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.218+0000 7f4bdc64e640 10 monclient: shutdown 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.218+0000 7f4bdc64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.218+0000 7f4bdc64e640 1 -- 172.21.15.64:0/385331619 >> v1:172.21.15.64:6789/0 conn(0x7f4bd40b17d0 legacy=0x7f4bd40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.219+0000 7f4bdc64e640 1 -- 172.21.15.64:0/385331619 shutdown_connections 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.219+0000 7f4bdc64e640 1 -- 172.21.15.64:0/385331619 >> 172.21.15.64:0/385331619 conn(0x7f4bd4066a00 msgr2=0x7f4bd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.219+0000 7f4bdc64e640 1 -- 172.21.15.64:0/385331619 shutdown_connections 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.219+0000 7f4bdc64e640 1 -- 172.21.15.64:0/385331619 wait complete. 2024-03-29T11:33:44.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 10 monclient: build_initial_monmap 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 10 monclient: monmap: 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:44.227 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 1 librados: starting msgr at 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 1 librados: starting objecter 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 1 Processor -- start 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 1 -- start start 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 1 librados: setting wanted keys 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 1 librados: calling monclient init 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.220+0000 7f4bdc64e640 10 monclient: init 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 10 monclient(hunting): picked mon.noname-b con 0x7f4bd40b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 10 monclient(hunting): picked mon.noname-a con 0x7f4bd41422b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 10 monclient(hunting): picked mon.noname-c con 0x7f4bd4142690 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdabeb640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4bd4142690 0x7f4bd41e7150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38030/0 (socket says 172.21.15.64:38030) 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdabeb640 1 -- 172.21.15.64:0/3034003285 learned_addr learned my addr 172.21.15.64:0/3034003285 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:44.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd41325f0 con 0x7f4bd41422b0 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.221+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd4131540 con 0x7f4bd40b17d0 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd41316d0 con 0x7f4bd4142690 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bdc64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bdc64e640 10 monclient(hunting): authenticate will time out at 1760.491986s 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 10 client.?.objecter ms_handle_connect 0x7f4bd4142690 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914618712 0 0) 0x7f4bd41316d0 con 0x7f4bd4142690 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.222+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb8004eb0 con 0x7f4bd4142690 2024-03-29T11:33:44.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 client.?.objecter ms_handle_connect 0x7f4bd40b17d0 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 client.?.objecter ms_handle_connect 0x7f4bd41422b0 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412617755 0 0) 0x7f4bd41325f0 con 0x7f4bd41422b0 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd41316d0 con 0x7f4bd41422b0 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590150794 0 0) 0x7f4bd4131540 con 0x7f4bd40b17d0 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:44.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd41325f0 con 0x7f4bd40b17d0 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282222953 0 0) 0x7f4bd41325f0 con 0x7f4bd40b17d0 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bd4131540 con 0x7f4bd40b17d0 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4bd0001830 con 0x7f4bd40b17d0 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): dump: 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.231 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853196799 0 0) 0x7f4bb8004eb0 con 0x7f4bd4142690 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bd41325f0 con 0x7f4bd4142690 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4bcc001e80 con 0x7f4bd4142690 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): dump: 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.232 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1334322256 0 0) 0x7f4bd41316d0 con 0x7f4bd41422b0 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bb8004eb0 con 0x7f4bd41422b0 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4bc40017d0 con 0x7f4bd41422b0 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.223+0000 7f4bcaffd640 10 monclient(hunting): dump: 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.233 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2970516059 0 0) 0x7f4bd41325f0 con 0x7f4bd4142690 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 >> v1:172.21.15.136:6789/0 conn(0x7f4bd40b17d0 legacy=0x7f4bd412bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 >> v1:172.21.15.64:6789/0 conn(0x7f4bd41422b0 legacy=0x7f4bd41dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 1 monclient: found mon.c 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:44.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd412c9d0 con 0x7f4bd4142690 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 10 monclient: _finish_auth 0 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 10 monclient: _renew_subs 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd412cc80 con 0x7f4bd4142690 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 10 monclient: _renew_subs 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd40b76b0 con 0x7f4bd4142690 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.224+0000 7f4bdc64e640 1 librados: init done 2024-03-29T11:33:44.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bcc001550 con 0x7f4bd4142690 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4bcc004fb0 con 0x7f4bd4142690 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcaffd640 10 monclient: dump: 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:44.236 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.225+0000 7f4bcbfff640 4 set_mon_vals no callback set 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.226+0000 7f4bdc64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.226+0000 7f4bdc64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.226+0000 7f4bdc64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.226+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc005080 con 0x7f4bd4142690 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.227+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4bcc01af80 con 0x7f4bd4142690 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f4bcc084e30 con 0x7f4bd4142690 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bcaffd640 10 client.4592.objecter ms_dispatch 0x7f4bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bcaffd640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:44.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bcaffd640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:44.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bcaffd640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:33:44.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bda3ea640 10 monclient: get_auth_request con 0x7f4bb806e190 auth_method 0 2024-03-29T11:33:44.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.228+0000 7f4bcaffd640 10 client.4592.objecter ms_handle_connect 0x7f4bb806e190 2024-03-29T11:33:44.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.238+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bcc050710 con 0x7f4bd4142690 2024-03-29T11:33:44.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.238+0000 7f4bcaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:44.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.238+0000 7f4bcaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:44.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.572+0000 7f4bdc64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:44.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.572+0000 7f4bdc64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:44.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.572+0000 7f4bdc64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:44.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.572+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bbc005330 con 0x7f4bd4142690 2024-03-29T11:33:44.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.573+0000 7f4bcaffd640 1 -- 172.21.15.64:0/3034003285 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f4bcc052640 con 0x7f4bd4142690 2024-03-29T11:33:44.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.573+0000 7f4bcaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:44.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.573+0000 7f4bcaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:44.574 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:44.574 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 10 librados: watch_flush enter 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 10 librados: watch_flush exit 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 >> v1:172.21.15.136:6816/2312274743 conn(0x7f4bb806e190 legacy=0x7f4bb8070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 10 monclient: shutdown 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 >> v1:172.21.15.64:6790/0 conn(0x7f4bd4142690 legacy=0x7f4bd41e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 shutdown_connections 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 >> 172.21.15.64:0/3034003285 conn(0x7f4bd4066a00 msgr2=0x7f4bd413db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 shutdown_connections 2024-03-29T11:33:44.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 -- 172.21.15.64:0/3034003285 wait complete. 2024-03-29T11:33:44.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:44.576+0000 7f4bdc64e640 1 librados: shutdown 2024-03-29T11:33:44.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:51.397 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:51.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.553+0000 7f124280f640 10 monclient: get_monmap_and_config 2024-03-29T11:33:51.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.553+0000 7f124280f640 10 monclient: build_initial_monmap 2024-03-29T11:33:51.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.553+0000 7f124280f640 10 monclient: monmap: 2024-03-29T11:33:51.552 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:51.552 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.553 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:51.555838+0000 2024-03-29T11:33:51.553 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:51.555838+0000 2024-03-29T11:33:51.553 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:51.553 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.553 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:51.554 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:51.554 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:51.554 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.554+0000 7f124280f640 1 Processor -- start 2024-03-29T11:33:51.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 1 -- start start 2024-03-29T11:33:51.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 10 monclient: init 2024-03-29T11:33:51.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:51.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:51.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 10 monclient(hunting): picked mon.noname-b con 0x7f123c1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:51.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 10 monclient(hunting): picked mon.noname-c con 0x7f123c13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:51.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.555+0000 7f124280f640 10 monclient(hunting): picked mon.noname-a con 0x7f123c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f124280f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c0aaef0 con 0x7f123c0a9d70 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f124280f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c145bd0 con 0x7f123c13f650 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f124280f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c145eb0 con 0x7f123c1457f0 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f124280f640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f124280f640 10 monclient(hunting): authenticate will time out at 1767.827038s 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f123c1457f0 0x7f123c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:57430/0 (socket says 172.21.15.64:57430) 2024-03-29T11:33:51.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123bfff640 1 -- 172.21.15.64:0/2389809679 learned_addr learned my addr 172.21.15.64:0/2389809679 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:51.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794848896 0 0) 0x7f123c0aaef0 con 0x7f123c0a9d70 2024-03-29T11:33:51.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:51.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:51.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:51.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:33:51.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1224003520 con 0x7f123c0a9d70 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356484332 0 0) 0x7f123c145eb0 con 0x7f123c1457f0 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.556+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c0aaef0 con 0x7f123c1457f0 2024-03-29T11:33:51.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3629621297 0 0) 0x7f123c0aaef0 con 0x7f123c1457f0 2024-03-29T11:33:51.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 >> v1:172.21.15.136:6789/0 conn(0x7f123c13f650 legacy=0x7f123c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 >> v1:172.21.15.64:6789/0 conn(0x7f123c0a9d70 legacy=0x7f123c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:51.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 monclient: found mon.noname-b 2024-03-29T11:33:51.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c146340 con 0x7f123c1457f0 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: _finish_auth 0 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1230001670 con 0x7f123c1457f0 2024-03-29T11:33:51.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:51.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: dump: 2024-03-29T11:33:51.567 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:51.567 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.567 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.567 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f124280f640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f124280f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12300017d0 con 0x7f123c1457f0 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 1 -- 172.21.15.64:0/2389809679 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1230001b60 con 0x7f123c1457f0 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f123affd640 10 monclient: dump: 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.569 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f124280f640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.557+0000 7f124280f640 4 set_mon_vals no callback set 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.558+0000 7f124280f640 10 monclient: shutdown 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.558+0000 7f124280f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.558+0000 7f124280f640 1 -- 172.21.15.64:0/2389809679 >> v1:172.21.15.64:6790/0 conn(0x7f123c1457f0 legacy=0x7f123c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.558+0000 7f124280f640 1 -- 172.21.15.64:0/2389809679 shutdown_connections 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.558+0000 7f124280f640 1 -- 172.21.15.64:0/2389809679 >> 172.21.15.64:0/2389809679 conn(0x7f123c066a00 msgr2=0x7f123c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.559+0000 7f124280f640 1 -- 172.21.15.64:0/2389809679 shutdown_connections 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.559+0000 7f124280f640 1 -- 172.21.15.64:0/2389809679 wait complete. 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.560+0000 7f124280f640 10 monclient: build_initial_monmap 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.560+0000 7f124280f640 10 monclient: monmap: 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:51.571 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.561+0000 7f124280f640 1 librados: starting msgr at 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.561+0000 7f124280f640 1 librados: starting objecter 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.561+0000 7f124280f640 1 Processor -- start 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.562+0000 7f124280f640 1 -- start start 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.562+0000 7f124280f640 1 librados: setting wanted keys 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.562+0000 7f124280f640 1 librados: calling monclient init 2024-03-29T11:33:51.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.562+0000 7f124280f640 10 monclient: init 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.563+0000 7f124280f640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.563+0000 7f124280f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.563+0000 7f124280f640 10 monclient(hunting): picked mon.noname-c con 0x7f123c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.563+0000 7f124280f640 10 monclient(hunting): picked mon.noname-a con 0x7f123c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f124280f640 10 monclient(hunting): picked mon.noname-b con 0x7f123c1457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f1240dac640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f123c1457f0 0x7f123c13a130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:51418/0 (socket says 172.21.15.64:51418) 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f1240dac640 1 -- 172.21.15.64:0/373544889 learned_addr learned my addr 172.21.15.64:0/373544889 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c13a840 con 0x7f123c13f650 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c071a70 con 0x7f123c1457f0 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f1238ff9640 10 client.?.objecter ms_handle_connect 0x7f123c1457f0 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f1238ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c070200 con 0x7f123c0a9d70 2024-03-29T11:33:51.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f124280f640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f124280f640 10 monclient(hunting): authenticate will time out at 1767.835585s 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.564+0000 7f123b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f123c13f650 0x7f123c1dec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59968/0 (socket says 172.21.15.64:59968) 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 client.?.objecter ms_handle_connect 0x7f123c13f650 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 client.?.objecter ms_handle_connect 0x7f123c0a9d70 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940064235 0 0) 0x7f123c13a840 con 0x7f123c13f650 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1228004eb0 con 0x7f123c13f650 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616685167 0 0) 0x7f123c071a70 con 0x7f123c1457f0 2024-03-29T11:33:51.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.565+0000 7f1238ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c13a840 con 0x7f123c1457f0 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084543691 0 0) 0x7f123c070200 con 0x7f123c0a9d70 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c071a70 con 0x7f123c0a9d70 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387248273 0 0) 0x7f123c13a840 con 0x7f123c1457f0 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.566+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f123c070200 con 0x7f123c1457f0 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1234001880 con 0x7f123c1457f0 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507623920 0 0) 0x7f123c071a70 con 0x7f123c0a9d70 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f123c13a840 con 0x7f123c0a9d70 2024-03-29T11:33:51.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551548238 0 0) 0x7f1228004eb0 con 0x7f123c13f650 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f123c071a70 con 0x7f123c13f650 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f1230001ae0 con 0x7f123c0a9d70 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:51.586 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f122c001c70 con 0x7f123c13f650 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:51.587 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 258320009 0 0) 0x7f123c070200 con 0x7f123c1457f0 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 >> v1:172.21.15.64:6790/0 conn(0x7f123c0a9d70 legacy=0x7f123c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 >> v1:172.21.15.64:6789/0 conn(0x7f123c13f650 legacy=0x7f123c1dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 monclient: found mon.b 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c06fbb0 con 0x7f123c1457f0 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient: _finish_auth 0 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f1238ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f124280f640 5 monclient: authenticate success, global_id 4606 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f124280f640 10 monclient: _renew_subs 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f124280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:51.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c06fe00 con 0x7f123c1457f0 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f124280f640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.567+0000 7f124280f640 10 monclient: _renew_subs 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1234003610 con 0x7f123c1457f0 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f124280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c070900 con 0x7f123c1457f0 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f124280f640 1 librados: init done 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f1238ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f12340039e0 con 0x7f123c1457f0 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f1238ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f1238ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f1238ff9640 10 monclient: dump: 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:51.589 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.568+0000 7f123a7fc640 4 set_mon_vals no callback set 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.569+0000 7f124280f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.569+0000 7f124280f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.569+0000 7f124280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.569+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123c070390 con 0x7f123c1457f0 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.570+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f1234003b40 con 0x7f123c1457f0 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.571+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7f1234084990 con 0x7f123c1457f0 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.571+0000 7f1238ff9640 10 client.4606.objecter ms_dispatch 0x7f123c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:51.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.571+0000 7f1238ff9640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.571+0000 7f1238ff9640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.571+0000 7f1238ff9640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.571+0000 7f123bfff640 10 monclient: get_auth_request con 0x7f122806e220 auth_method 0 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.572+0000 7f1238ff9640 10 client.4606.objecter ms_handle_connect 0x7f122806e220 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.580+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1234050270 con 0x7f123c1457f0 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.580+0000 7f1238ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:51.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.580+0000 7f1238ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:51.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.964+0000 7f124280f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:51.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.964+0000 7f124280f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:51.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.964+0000 7f124280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:33:51.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.964+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f123c075f10 con 0x7f123c1457f0 2024-03-29T11:33:51.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.965+0000 7f1238ff9640 1 -- 172.21.15.64:0/373544889 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7f12340521a0 con 0x7f123c1457f0 2024-03-29T11:33:51.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.965+0000 7f1238ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:51.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.965+0000 7f1238ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:51.964 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:51.964 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:51.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 10 librados: watch_flush enter 2024-03-29T11:33:51.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 10 librados: watch_flush exit 2024-03-29T11:33:51.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 10 client.4606.objecter successfully canceled tick 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 >> v1:172.21.15.136:6816/2312274743 conn(0x7f122806e220 legacy=0x7f12280706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 10 monclient: shutdown 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 >> v1:172.21.15.136:6789/0 conn(0x7f123c1457f0 legacy=0x7f123c13a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 shutdown_connections 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.969+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 >> 172.21.15.64:0/373544889 conn(0x7f123c066a00 msgr2=0x7f123c13ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.970+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 shutdown_connections 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.970+0000 7f124280f640 1 -- 172.21.15.64:0/373544889 wait complete. 2024-03-29T11:33:51.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:51.970+0000 7f124280f640 1 librados: shutdown 2024-03-29T11:33:51.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:58.790 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:58.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.938+0000 7fbcedc58640 10 monclient: get_monmap_and_config 2024-03-29T11:33:58.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.938+0000 7fbcedc58640 10 monclient: build_initial_monmap 2024-03-29T11:33:58.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.938+0000 7fbcedc58640 10 monclient: monmap: 2024-03-29T11:33:58.935 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:33:58.939517+0000 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:33:58.939517+0000 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:33:58.936 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:58.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.939+0000 7fbcedc58640 1 Processor -- start 2024-03-29T11:33:58.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.939+0000 7fbcedc58640 1 -- start start 2024-03-29T11:33:58.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.939+0000 7fbcedc58640 10 monclient: init 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient(hunting): picked mon.noname-b con 0x7fbce00389e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient(hunting): picked mon.noname-c con 0x7fbce0038dc0 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient(hunting): picked mon.noname-a con 0x7fbce0030660 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce00c54c0 con 0x7fbce0030660 2024-03-29T11:33:58.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce0036430 con 0x7fbce0038dc0 2024-03-29T11:33:58.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce0039840 con 0x7fbce00389e0 2024-03-29T11:33:58.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:58.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.940+0000 7fbcedc58640 10 monclient(hunting): authenticate will time out at 1775.210687s 2024-03-29T11:33:58.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.942+0000 7fbcecc56640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbce00389e0 0x7fbce0033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34660/0 (socket says 172.21.15.64:34660) 2024-03-29T11:33:58.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.942+0000 7fbcecc56640 1 -- 172.21.15.64:0/1204691338 learned_addr learned my addr 172.21.15.64:0/1204691338 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:58.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.944+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368301450 0 0) 0x7fbce0039840 con 0x7fbce00389e0 2024-03-29T11:33:58.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.944+0000 7fbce77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.944+0000 7fbce77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.944+0000 7fbce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.944+0000 7fbce77fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.944+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8003520 con 0x7fbce00389e0 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777985873 0 0) 0x7fbcd8003520 con 0x7fbce00389e0 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 >> v1:172.21.15.136:6789/0 conn(0x7fbce0038dc0 legacy=0x7fbce00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 >> v1:172.21.15.64:6789/0 conn(0x7fbce0030660 legacy=0x7fbce00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:58.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 monclient: found mon.noname-b 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce0035690 con 0x7fbce00389e0 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbcedc58640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbcedc58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:58.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fbcdc001690 con 0x7fbce00389e0 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: dump: 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:58.945 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:58.946 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:58.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbcedc58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:58.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcdc001920 con 0x7fbce00389e0 2024-03-29T11:33:58.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbce77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:58.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbcedc58640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:58.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.945+0000 7fbcedc58640 4 set_mon_vals no callback set 2024-03-29T11:33:58.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.946+0000 7fbce77fe640 1 -- 172.21.15.64:0/1204691338 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fbcdc001cb0 con 0x7fbce00389e0 2024-03-29T11:33:58.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.946+0000 7fbce77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.946+0000 7fbce77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:58.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.946+0000 7fbce77fe640 10 monclient: dump: 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.947+0000 7fbcedc58640 10 monclient: shutdown 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.947+0000 7fbcedc58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:58.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.947+0000 7fbcedc58640 1 -- 172.21.15.64:0/1204691338 >> v1:172.21.15.64:6790/0 conn(0x7fbce00389e0 legacy=0x7fbce0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.949+0000 7fbcedc58640 1 -- 172.21.15.64:0/1204691338 shutdown_connections 2024-03-29T11:33:58.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.949+0000 7fbcedc58640 1 -- 172.21.15.64:0/1204691338 >> 172.21.15.64:0/1204691338 conn(0x7fbce0005c00 msgr2=0x7fbce0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:58.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.950+0000 7fbcedc58640 1 -- 172.21.15.64:0/1204691338 shutdown_connections 2024-03-29T11:33:58.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.950+0000 7fbcedc58640 1 -- 172.21.15.64:0/1204691338 wait complete. 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.950+0000 7fbcedc58640 10 monclient: build_initial_monmap 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.950+0000 7fbcedc58640 10 monclient: monmap: 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:33:58.950 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 1 librados: starting msgr at 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 1 librados: starting objecter 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 1 Processor -- start 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 1 -- start start 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 1 librados: setting wanted keys 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 1 librados: calling monclient init 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.951+0000 7fbcedc58640 10 monclient: init 2024-03-29T11:33:58.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient(hunting): picked mon.noname-b con 0x7fbce0030660 addr v1:172.21.15.136:6789/0 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient(hunting): picked mon.noname-c con 0x7fbce00389e0 addr v1:172.21.15.64:6790/0 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient(hunting): picked mon.noname-a con 0x7fbce002ff20 addr v1:172.21.15.64:6789/0 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce0032cd0 con 0x7fbce002ff20 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce00ca850 con 0x7fbce0030660 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce00ca1c0 con 0x7fbce00389e0 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:58.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.952+0000 7fbcedc58640 10 monclient(hunting): authenticate will time out at 1775.222604s 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.955+0000 7fbce7fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbce00389e0 0x7fbce00be170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35084/0 (socket says 172.21.15.64:35084) 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.955+0000 7fbce7fff640 1 -- 172.21.15.64:0/83757136 learned_addr learned my addr 172.21.15.64:0/83757136 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 client.?.objecter ms_handle_connect 0x7fbce00389e0 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 client.?.objecter ms_handle_connect 0x7fbce0030660 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:58.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524927829 0 0) 0x7fbce00ca1c0 con 0x7fbce00389e0 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbccc004eb0 con 0x7fbce00389e0 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 462547996 0 0) 0x7fbccc004eb0 con 0x7fbce00389e0 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbce00ca1c0 con 0x7fbce00389e0 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fbcd4004080 con 0x7fbce00389e0 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): dump: 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:58.954 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:58.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.956+0000 7fbce57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085563947 0 0) 0x7fbce00ca850 con 0x7fbce0030660 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbccc005680 con 0x7fbce0030660 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3575999319 0 0) 0x7fbce00ca1c0 con 0x7fbce00389e0 2024-03-29T11:33:58.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 >> v1:172.21.15.136:6789/0 conn(0x7fbce0030660 legacy=0x7fbce00c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.957+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 >> v1:172.21.15.64:6789/0 conn(0x7fbce002ff20 legacy=0x7fbce01639b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 1 monclient: found mon.c 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce00ca350 con 0x7fbce00389e0 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 10 monclient: _finish_auth 0 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbce57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 10 monclient: _renew_subs 2024-03-29T11:33:58.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce00c82c0 con 0x7fbce00389e0 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 10 monclient: _renew_subs 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce00be880 con 0x7fbce00389e0 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.958+0000 7fbcedc58640 1 librados: init done 2024-03-29T11:33:58.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.959+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcd4000ed0 con 0x7fbce00389e0 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.959+0000 7fbce57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbcedc58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbcedc58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbcedc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbce6ffd640 4 set_mon_vals no callback set 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce0007b90 con 0x7fbce00389e0 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fbcd4004080 con 0x7fbce00389e0 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbce57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbce57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.960+0000 7fbce57fa640 10 monclient: dump: 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:33:58.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.962+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fbcd4001140 con 0x7fbce00389e0 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbcecc56640 10 monclient: get_auth_request con 0x7fbccc06e2e0 auth_method 0 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3340349273 0 0) 0x7fbcd40544d0 con 0x7fbce00389e0 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbce57fa640 10 client.4604.objecter ms_dispatch 0x7fbce8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbce57fa640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbce57fa640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbce57fa640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:33:58.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.963+0000 7fbce57fa640 10 client.4604.objecter ms_handle_connect 0x7fbccc06e2e0 2024-03-29T11:33:58.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.972+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbcd4050770 con 0x7fbce00389e0 2024-03-29T11:33:58.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.972+0000 7fbce57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:58.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:58.972+0000 7fbce57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:59.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.302+0000 7fbcedc58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:59.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.302+0000 7fbcedc58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:59.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.302+0000 7fbcedc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:33:59.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.302+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbce0005120 con 0x7fbce00389e0 2024-03-29T11:33:59.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.304+0000 7fbce57fa640 1 -- 172.21.15.64:0/83757136 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1633336633) 0x7fbcd4050910 con 0x7fbce00389e0 2024-03-29T11:33:59.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.304+0000 7fbce57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:59.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.304+0000 7fbce57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:59.301 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:33:59.301 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:24:02.042497+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:02.042497+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 10 librados: watch_flush enter 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 10 librados: watch_flush exit 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 >> v1:172.21.15.136:6816/2312274743 conn(0x7fbccc06e2e0 legacy=0x7fbccc0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 10 monclient: shutdown 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:59.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.307+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 >> v1:172.21.15.64:6790/0 conn(0x7fbce00389e0 legacy=0x7fbce00be170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:59.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.308+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 shutdown_connections 2024-03-29T11:33:59.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.308+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 >> 172.21.15.64:0/83757136 conn(0x7fbce0005c00 msgr2=0x7fbce00c2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:59.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.308+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 shutdown_connections 2024-03-29T11:33:59.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.308+0000 7fbcedc58640 1 -- 172.21.15.64:0/83757136 wait complete. 2024-03-29T11:33:59.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:33:59.308+0000 7fbcedc58640 1 librados: shutdown 2024-03-29T11:33:59.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:06.131 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.289+0000 7f693ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.289+0000 7f693ffff640 10 monclient: build_initial_monmap 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.289+0000 7f693ffff640 10 monclient: monmap: 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:06.290229+0000 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:06.290229+0000 2024-03-29T11:34:06.287 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.290+0000 7f693ffff640 1 Processor -- start 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.290+0000 7f693ffff640 1 -- start start 2024-03-29T11:34:06.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.290+0000 7f693ffff640 10 monclient: init 2024-03-29T11:34:06.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:06.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6938034850 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6938006410 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f693802a640 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693802c820 con 0x7f693802a640 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693802a120 con 0x7f6938006410 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380c75b0 con 0x7f6938034850 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.291+0000 7f693ffff640 10 monclient(hunting): authenticate will time out at 1782.561165s 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.292+0000 7f693effd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6938034850 0x7f69380c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58456/0 (socket says 172.21.15.64:58456) 2024-03-29T11:34:06.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.292+0000 7f693effd640 1 -- 172.21.15.64:0/367385356 learned_addr learned my addr 172.21.15.64:0/367385356 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082253075 0 0) 0x7f69380c75b0 con 0x7f6938034850 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c005730 con 0x7f6938034850 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606195650 0 0) 0x7f693802c820 con 0x7f693802a640 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:06.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:06.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:06.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:34:06.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.294+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69380c75b0 con 0x7f693802a640 2024-03-29T11:34:06.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455263356 0 0) 0x7f69380c75b0 con 0x7f693802a640 2024-03-29T11:34:06.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 >> v1:172.21.15.64:6790/0 conn(0x7f6938034850 legacy=0x7f69380c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 >> v1:172.21.15.136:6789/0 conn(0x7f6938006410 legacy=0x7f69380cc220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:06.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:06.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 1 monclient: found mon.noname-a 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938033110 con 0x7f693802a640 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f694004f730 con 0x7f693802a640 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.295+0000 7f693dffb640 10 monclient: dump: 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.295 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.296+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69400008d0 con 0x7f693802a640 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.296+0000 7f693dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.296+0000 7f693dffb640 1 -- 172.21.15.64:0/367385356 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f69400758b0 con 0x7f693802a640 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.296+0000 7f693dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.296+0000 7f693dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.296+0000 7f693dffb640 10 monclient: dump: 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:06.296 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:06.297 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.298+0000 7f693ffff640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.298+0000 7f693ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.298+0000 7f693ffff640 4 set_mon_vals no callback set 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.299+0000 7f693ffff640 10 monclient: shutdown 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.299+0000 7f693ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.299+0000 7f693ffff640 1 -- 172.21.15.64:0/367385356 >> v1:172.21.15.64:6789/0 conn(0x7f693802a640 legacy=0x7f69380320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.299+0000 7f693ffff640 1 -- 172.21.15.64:0/367385356 shutdown_connections 2024-03-29T11:34:06.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.299+0000 7f693ffff640 1 -- 172.21.15.64:0/367385356 >> 172.21.15.64:0/367385356 conn(0x7f69380058f0 msgr2=0x7f6938005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.300+0000 7f693ffff640 1 -- 172.21.15.64:0/367385356 shutdown_connections 2024-03-29T11:34:06.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.300+0000 7f693ffff640 1 -- 172.21.15.64:0/367385356 wait complete. 2024-03-29T11:34:06.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.301+0000 7f693ffff640 10 monclient: build_initial_monmap 2024-03-29T11:34:06.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.301+0000 7f693ffff640 10 monclient: monmap: 2024-03-29T11:34:06.300 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:06.300 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:06.301 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:06.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 1 librados: starting msgr at 2024-03-29T11:34:06.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 1 librados: starting objecter 2024-03-29T11:34:06.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 1 Processor -- start 2024-03-29T11:34:06.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 1 -- start start 2024-03-29T11:34:06.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 1 librados: setting wanted keys 2024-03-29T11:34:06.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 1 librados: calling monclient init 2024-03-29T11:34:06.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.302+0000 7f693ffff640 10 monclient: init 2024-03-29T11:34:06.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.303+0000 7f693ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:06.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.303+0000 7f693ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:06.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.303+0000 7f693ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6938006410 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:06.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.303+0000 7f693ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f693802a640 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:06.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.304+0000 7f693ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6938034850 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:06.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.304+0000 7f693ffff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380c03f0 con 0x7f6938006410 2024-03-29T11:34:06.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.304+0000 7f693ffff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380c0110 con 0x7f6938034850 2024-03-29T11:34:06.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.304+0000 7f693ffff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380bf130 con 0x7f693802a640 2024-03-29T11:34:06.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.304+0000 7f693ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:06.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.304+0000 7f693ffff640 10 monclient(hunting): authenticate will time out at 1782.574725s 2024-03-29T11:34:06.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.306+0000 7f693e7fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f693802a640 0x7f69380bae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58972/0 (socket says 172.21.15.64:58972) 2024-03-29T11:34:06.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.306+0000 7f693e7fc640 1 -- 172.21.15.64:0/1003407001 learned_addr learned my addr 172.21.15.64:0/1003407001 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:06.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.307+0000 7f6937fff640 10 client.?.objecter ms_handle_connect 0x7f6938034850 2024-03-29T11:34:06.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.307+0000 7f6937fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:06.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.307+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617426226 0 0) 0x7f69380c0110 con 0x7f6938034850 2024-03-29T11:34:06.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.307+0000 7f6937fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:06.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.307+0000 7f6937fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:06.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:06.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:34:06.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920004eb0 con 0x7f6938034850 2024-03-29T11:34:06.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 client.?.objecter ms_handle_connect 0x7f693802a640 2024-03-29T11:34:06.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:06.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1634268152 0 0) 0x7f6920004eb0 con 0x7f6938034850 2024-03-29T11:34:06.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69380c0110 con 0x7f6938034850 2024-03-29T11:34:06.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f694004f400 con 0x7f6938034850 2024-03-29T11:34:06.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:06.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:06.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): dump: 2024-03-29T11:34:06.320 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:06.320 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:06.320 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.321 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.321 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:06.321 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:06.321 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:06.321 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:06.321 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:06.322 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:06.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:06.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310773449 0 0) 0x7f69380bf130 con 0x7f693802a640 2024-03-29T11:34:06.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:06.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:06.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:06.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:34:06.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.308+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920004eb0 con 0x7f693802a640 2024-03-29T11:34:06.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080170239 0 0) 0x7f69380c0110 con 0x7f6938034850 2024-03-29T11:34:06.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 >> v1:172.21.15.64:6790/0 conn(0x7f693802a640 legacy=0x7f69380bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 >> v1:172.21.15.64:6789/0 conn(0x7f6938006410 legacy=0x7f69380bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:06.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 1 monclient: found mon.b 2024-03-29T11:34:06.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:06.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:06.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69380bf2c0 con 0x7f6938034850 2024-03-29T11:34:06.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:06.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:06.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f6937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:06.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 5 monclient: authenticate success, global_id 4612 2024-03-29T11:34:06.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 10 monclient: _renew_subs 2024-03-29T11:34:06.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:06.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69380bb520 con 0x7f6938034850 2024-03-29T11:34:06.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:06.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 10 monclient: _renew_subs 2024-03-29T11:34:06.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:06.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69380c38f0 con 0x7f6938034850 2024-03-29T11:34:06.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.309+0000 7f693ffff640 1 librados: init done 2024-03-29T11:34:06.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.310+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f694004fdd0 con 0x7f6938034850 2024-03-29T11:34:06.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.310+0000 7f6937fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:06.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.310+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f6940067bf0 con 0x7f6938034850 2024-03-29T11:34:06.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.310+0000 7f6937fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:06.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.310+0000 7f6937fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:06.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.310+0000 7f6937fff640 10 monclient: dump: 2024-03-29T11:34:06.327 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:06.328 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:06.328 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.328 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:06.328 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:06.328 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:06.328 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:06.329 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:06.329 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:06.329 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:06.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.311+0000 7f693d7fa640 4 set_mon_vals no callback set 2024-03-29T11:34:06.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.313+0000 7f693ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:06.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.313+0000 7f693ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:06.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.314+0000 7f693ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:06.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.314+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6924005080 con 0x7f6938034850 2024-03-29T11:34:06.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.315+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f6940067d50 con 0x7f6938034850 2024-03-29T11:34:06.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.315+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7f69400fe420 con 0x7f6938034850 2024-03-29T11:34:06.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.315+0000 7f6937fff640 10 client.4612.objecter ms_dispatch 0x7f6940000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:06.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.315+0000 7f6937fff640 3 client.4612.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:06.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.315+0000 7f6937fff640 3 client.4612.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:06.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.315+0000 7f6937fff640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-29T11:34:06.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.317+0000 7f693effd640 10 monclient: get_auth_request con 0x7f692006d920 auth_method 0 2024-03-29T11:34:06.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.317+0000 7f6937fff640 10 client.4612.objecter ms_handle_connect 0x7f692006d920 2024-03-29T11:34:06.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.325+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69400c9d00 con 0x7f6938034850 2024-03-29T11:34:06.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.325+0000 7f6937fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:06.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.326+0000 7f6937fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:06.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.660+0000 7f693ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:06.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.660+0000 7f693ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:06.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.660+0000 7f693ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:06.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.660+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6924005330 con 0x7f6938034850 2024-03-29T11:34:06.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.663+0000 7f6937fff640 1 -- 172.21.15.64:0/1003407001 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7f69400cbc30 con 0x7f6938034850 2024-03-29T11:34:06.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.663+0000 7f6937fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:06.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.663+0000 7f6937fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:06.662 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:06.662 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:06.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 10 librados: watch_flush enter 2024-03-29T11:34:06.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 10 librados: watch_flush exit 2024-03-29T11:34:06.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 10 client.4612.objecter successfully canceled tick 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 >> v1:172.21.15.136:6816/2312274743 conn(0x7f692006d920 legacy=0x7f692006fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 10 monclient: shutdown 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.668+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 >> v1:172.21.15.136:6789/0 conn(0x7f6938034850 legacy=0x7f69381586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.669+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 shutdown_connections 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.669+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 >> 172.21.15.64:0/1003407001 conn(0x7f69380058f0 msgr2=0x7f69380043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.670+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 shutdown_connections 2024-03-29T11:34:06.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.670+0000 7f693ffff640 1 -- 172.21.15.64:0/1003407001 wait complete. 2024-03-29T11:34:06.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:06.670+0000 7f693ffff640 1 librados: shutdown 2024-03-29T11:34:06.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:13.490 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:13.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.648+0000 7fa1b1532640 10 monclient: get_monmap_and_config 2024-03-29T11:34:13.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.648+0000 7fa1b1532640 10 monclient: build_initial_monmap 2024-03-29T11:34:13.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.648+0000 7fa1b1532640 10 monclient: monmap: 2024-03-29T11:34:13.645 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:13.645 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:13.645 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:13.649369+0000 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:13.649369+0000 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:13.646 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:13.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.651+0000 7fa1b1532640 1 Processor -- start 2024-03-29T11:34:13.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.651+0000 7fa1b1532640 1 -- start start 2024-03-29T11:34:13.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.651+0000 7fa1b1532640 10 monclient: init 2024-03-29T11:34:13.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.652+0000 7fa1b1532640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 10 monclient(hunting): picked mon.noname-b con 0x7fa1ac1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 10 monclient(hunting): picked mon.noname-c con 0x7fa1ac13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:13.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 10 monclient(hunting): picked mon.noname-a con 0x7fa1ac0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:13.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0aaef0 con 0x7fa1ac0a9d70 2024-03-29T11:34:13.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac145bd0 con 0x7fa1ac13f650 2024-03-29T11:34:13.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac145eb0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:13.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.653+0000 7fa1b1532640 10 monclient(hunting): authenticate will time out at 1789.923364s 2024-03-29T11:34:13.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1aaffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fa1ac1457f0 0x7fa1ac13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37314/0 (socket says 172.21.15.64:37314) 2024-03-29T11:34:13.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1aaffd640 1 -- 172.21.15.64:0/139983284 learned_addr learned my addr 172.21.15.64:0/139983284 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:13.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521183574 0 0) 0x7fa1ac145eb0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1a9ffb640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:34:13.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.656+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa194003520 con 0x7fa1ac1457f0 2024-03-29T11:34:13.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.657+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657885632 0 0) 0x7fa194003520 con 0x7fa1ac1457f0 2024-03-29T11:34:13.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.657+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 >> v1:172.21.15.136:6789/0 conn(0x7fa1ac13f650 legacy=0x7fa1ac0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.659+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 >> v1:172.21.15.64:6789/0 conn(0x7fa1ac0a9d70 legacy=0x7fa1ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.659+0000 7fa1a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac146340 con 0x7fa1ac1457f0 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:13.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa1a0001690 con 0x7fa1ac1457f0 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: dump: 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:13.658 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:13.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1b1532640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:34:13.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1b1532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1a0001920 con 0x7fa1ac1457f0 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 1 -- 172.21.15.64:0/139983284 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa1a0001cb0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.660+0000 7fa1a9ffb640 10 monclient: dump: 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.660 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.661+0000 7fa1b1532640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.661+0000 7fa1b1532640 4 set_mon_vals no callback set 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 10 monclient: shutdown 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 1 -- 172.21.15.64:0/139983284 >> v1:172.21.15.64:6790/0 conn(0x7fa1ac1457f0 legacy=0x7fa1ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 1 -- 172.21.15.64:0/139983284 shutdown_connections 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 1 -- 172.21.15.64:0/139983284 >> 172.21.15.64:0/139983284 conn(0x7fa1ac066a00 msgr2=0x7fa1ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:13.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 1 -- 172.21.15.64:0/139983284 shutdown_connections 2024-03-29T11:34:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.662+0000 7fa1b1532640 1 -- 172.21.15.64:0/139983284 wait complete. 2024-03-29T11:34:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.663+0000 7fa1b1532640 10 monclient: build_initial_monmap 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.663+0000 7fa1b1532640 10 monclient: monmap: 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.664+0000 7fa1b1532640 1 librados: starting msgr at 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.664+0000 7fa1b1532640 1 librados: starting objecter 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.665+0000 7fa1b1532640 1 Processor -- start 2024-03-29T11:34:13.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.665+0000 7fa1b1532640 1 -- start start 2024-03-29T11:34:13.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.665+0000 7fa1b1532640 1 librados: setting wanted keys 2024-03-29T11:34:13.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.665+0000 7fa1b1532640 1 librados: calling monclient init 2024-03-29T11:34:13.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.665+0000 7fa1b1532640 10 monclient: init 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 10 monclient(hunting): picked mon.noname-a con 0x7fa1ac0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 10 monclient(hunting): picked mon.noname-b con 0x7fa1ac13f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 10 monclient(hunting): picked mon.noname-c con 0x7fa1ac1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0ac740 con 0x7fa1ac0a9d70 2024-03-29T11:34:13.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.667+0000 7fa1b1532640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac142460 con 0x7fa1ac13f650 2024-03-29T11:34:13.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.668+0000 7fa1b1532640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac142080 con 0x7fa1ac1457f0 2024-03-29T11:34:13.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.668+0000 7fa1b1532640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:13.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.668+0000 7fa1b1532640 10 monclient(hunting): authenticate will time out at 1789.938237s 2024-03-29T11:34:13.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.671+0000 7fa1ab7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fa1ac1457f0 0x7fa1ac1e1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38334/0 (socket says 172.21.15.64:38334) 2024-03-29T11:34:13.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.671+0000 7fa1ab7fe640 1 -- 172.21.15.64:0/1258145506 learned_addr learned my addr 172.21.15.64:0/1258145506 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:13.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.671+0000 7fa1937fe640 10 client.?.objecter ms_handle_connect 0x7fa1ac1457f0 2024-03-29T11:34:13.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.671+0000 7fa1937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041786465 0 0) 0x7fa1ac142080 con 0x7fa1ac1457f0 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa198004eb0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 client.?.objecter ms_handle_connect 0x7fa1ac0a9d70 2024-03-29T11:34:13.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719015792 0 0) 0x7fa1ac0ac740 con 0x7fa1ac0a9d70 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1980051d0 con 0x7fa1ac0a9d70 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 client.?.objecter ms_handle_connect 0x7fa1ac13f650 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.672+0000 7fa1937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 739835905 0 0) 0x7fa198004eb0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1ac0ac740 con 0x7fa1ac1457f0 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa19c0018f0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:13.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): dump: 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915979539 0 0) 0x7fa1ac142460 con 0x7fa1ac13f650 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 10 monclient(hunting): my global_id is 4615 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.673+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa198004eb0 con 0x7fa1ac13f650 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4117726049 0 0) 0x7fa1ac0ac740 con 0x7fa1ac1457f0 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 >> v1:172.21.15.136:6789/0 conn(0x7fa1ac13f650 legacy=0x7fa1ac141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 >> v1:172.21.15.64:6789/0 conn(0x7fa1ac0a9d70 legacy=0x7fa1ac13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 1 monclient: found mon.c 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac139730 con 0x7fa1ac1457f0 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 10 monclient: _renew_subs 2024-03-29T11:34:13.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:34:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac1399e0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 10 monclient: _renew_subs 2024-03-29T11:34:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:34:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac142b60 con 0x7fa1ac1457f0 2024-03-29T11:34:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.674+0000 7fa1b1532640 1 librados: init done 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa19c003610 con 0x7fa1ac1457f0 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa19c0039e0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1937fe640 10 monclient: dump: 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:13.678 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1b1532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1b1532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1b1532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac1425f0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.678+0000 7fa1a8ff9640 4 set_mon_vals no callback set 2024-03-29T11:34:13.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.680+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fa19c003bc0 con 0x7fa1ac1457f0 2024-03-29T11:34:13.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.681+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7fa19c003800 con 0x7fa1ac1457f0 2024-03-29T11:34:13.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.681+0000 7fa1937fe640 10 client.4616.objecter ms_dispatch 0x7fa1ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:13.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.681+0000 7fa1937fe640 3 client.4616.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:13.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.681+0000 7fa1937fe640 3 client.4616.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:13.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.681+0000 7fa1937fe640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T11:34:13.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.684+0000 7fa1aaffd640 10 monclient: get_auth_request con 0x7fa19806e310 auth_method 0 2024-03-29T11:34:13.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.684+0000 7fa1937fe640 10 client.4616.objecter ms_handle_connect 0x7fa19806e310 2024-03-29T11:34:13.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.691+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa19c050820 con 0x7fa1ac1457f0 2024-03-29T11:34:13.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.691+0000 7fa1937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:13.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:13.691+0000 7fa1937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:14.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.024+0000 7fa1b1532640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:14.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.024+0000 7fa1b1532640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:14.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.024+0000 7fa1b1532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T11:34:14.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.024+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1ac1430a0 con 0x7fa1ac1457f0 2024-03-29T11:34:14.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.027+0000 7fa1937fe640 1 -- 172.21.15.64:0/1258145506 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7fa19c0509c0 con 0x7fa1ac1457f0 2024-03-29T11:34:14.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.027+0000 7fa1937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:14.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.027+0000 7fa1937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:14.025 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:14.025 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:14.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.030+0000 7fa1b1532640 10 librados: watch_flush enter 2024-03-29T11:34:14.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 10 librados: watch_flush exit 2024-03-29T11:34:14.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T11:34:14.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 10 client.4616.objecter successfully canceled tick 2024-03-29T11:34:14.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 >> v1:172.21.15.136:6816/2312274743 conn(0x7fa19806e310 legacy=0x7fa1980707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 10 monclient: shutdown 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 >> v1:172.21.15.64:6790/0 conn(0x7fa1ac1457f0 legacy=0x7fa1ac1e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 shutdown_connections 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 >> 172.21.15.64:0/1258145506 conn(0x7fa1ac066a00 msgr2=0x7fa1ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 shutdown_connections 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 -- 172.21.15.64:0/1258145506 wait complete. 2024-03-29T11:34:14.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:14.031+0000 7fa1b1532640 1 librados: shutdown 2024-03-29T11:34:14.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:20.851 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.008+0000 7f40d1f3d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.008+0000 7f40d1f3d640 10 monclient: build_initial_monmap 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.008+0000 7f40d1f3d640 10 monclient: monmap: 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:21.009275+0000 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:21.009275+0000 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:21.006 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:21.007 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:21.007 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:21.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.009+0000 7f40d1f3d640 1 Processor -- start 2024-03-29T11:34:21.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.009+0000 7f40d1f3d640 1 -- start start 2024-03-29T11:34:21.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.009+0000 7f40d1f3d640 10 monclient: init 2024-03-29T11:34:21.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient(hunting): picked mon.noname-c con 0x7f40cc0af0f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient(hunting): picked mon.noname-a con 0x7f40cc0af4d0 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient(hunting): picked mon.noname-b con 0x7f40cc0ab170 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc13f010 con 0x7f40cc0af4d0 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc0ab6c0 con 0x7f40cc0af0f0 2024-03-29T11:34:21.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc0ab9a0 con 0x7f40cc0ab170 2024-03-29T11:34:21.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:21.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.010+0000 7f40d1f3d640 10 monclient(hunting): authenticate will time out at 1797.280720s 2024-03-29T11:34:21.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.014+0000 7f40cb7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f40cc0af0f0 0x7f40cc13c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:35865/0 (socket says 172.21.15.64:35865) 2024-03-29T11:34:21.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.014+0000 7f40cb7fe640 1 -- 172.21.15.64:0/2790312129 learned_addr learned my addr 172.21.15.64:0/2790312129 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.015+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266700259 0 0) 0x7f40cc13f010 con 0x7f40cc0af4d0 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.015+0000 7f40ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.015+0000 7f40ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.015+0000 7f40ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.015+0000 7f40ca7fc640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.015+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b4003520 con 0x7f40cc0af4d0 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740593260 0 0) 0x7f40b4003520 con 0x7f40cc0af4d0 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 >> v1:172.21.15.64:6790/0 conn(0x7f40cc0ab170 legacy=0x7f40cc14a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 >> v1:172.21.15.136:6789/0 conn(0x7f40cc0af0f0 legacy=0x7f40cc13c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 1 monclient: found mon.noname-a 2024-03-29T11:34:21.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0abc80 con 0x7f40cc0af4d0 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f40c0001690 con 0x7f40cc0af4d0 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.016+0000 7f40ca7fc640 10 monclient: dump: 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.017 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40c0001860 con 0x7f40cc0af4d0 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40ca7fc640 1 -- 172.21.15.64:0/2790312129 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f40c0001cb0 con 0x7f40cc0af4d0 2024-03-29T11:34:21.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40ca7fc640 10 monclient: dump: 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40d1f3d640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40d1f3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:21.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.017+0000 7f40d1f3d640 4 set_mon_vals no callback set 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 10 monclient: shutdown 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 1 -- 172.21.15.64:0/2790312129 >> v1:172.21.15.64:6789/0 conn(0x7f40cc0af4d0 legacy=0x7f40cc13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 1 -- 172.21.15.64:0/2790312129 shutdown_connections 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 1 -- 172.21.15.64:0/2790312129 >> 172.21.15.64:0/2790312129 conn(0x7f40cc0b35e0 msgr2=0x7f40cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 1 -- 172.21.15.64:0/2790312129 shutdown_connections 2024-03-29T11:34:21.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.018+0000 7f40d1f3d640 1 -- 172.21.15.64:0/2790312129 wait complete. 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.019+0000 7f40d1f3d640 10 monclient: build_initial_monmap 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.019+0000 7f40d1f3d640 10 monclient: monmap: 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:21.022 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.019+0000 7f40d1f3d640 1 librados: starting msgr at 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.019+0000 7f40d1f3d640 1 librados: starting objecter 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.020+0000 7f40d1f3d640 1 Processor -- start 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.020+0000 7f40d1f3d640 1 -- start start 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.020+0000 7f40d1f3d640 1 librados: setting wanted keys 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.020+0000 7f40d1f3d640 1 librados: calling monclient init 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.020+0000 7f40d1f3d640 10 monclient: init 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:21.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient(hunting): picked mon.noname-c con 0x7f40cc0ab170 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient(hunting): picked mon.noname-a con 0x7f40cc0af0f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient(hunting): picked mon.noname-b con 0x7f40cc11f1e0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc1367d0 con 0x7f40cc0af0f0 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc136960 con 0x7f40cc11f1e0 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40cc13b210 con 0x7f40cc0ab170 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.021+0000 7f40d1f3d640 10 monclient(hunting): authenticate will time out at 1797.291296s 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.024+0000 7f40cbfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f40cc11f1e0 0x7f40cc13ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:35911/0 (socket says 172.21.15.64:35911) 2024-03-29T11:34:21.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.025+0000 7f40cbfff640 1 -- 172.21.15.64:0/1983575427 learned_addr learned my addr 172.21.15.64:0/1983575427 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 10 client.?.objecter ms_handle_connect 0x7f40cc11f1e0 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469773423 0 0) 0x7f40cc136960 con 0x7f40cc11f1e0 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 10 monclient(hunting): my global_id is 4618 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b8004eb0 con 0x7f40cc11f1e0 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 662290927 0 0) 0x7f40b8004eb0 con 0x7f40cc11f1e0 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40cc136960 con 0x7f40cc11f1e0 2024-03-29T11:34:21.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.026+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f40c4001c60 con 0x7f40cc11f1e0 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 10 monclient(hunting): dump: 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:21.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4092075170 0 0) 0x7f40cc136960 con 0x7f40cc11f1e0 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 >> v1:172.21.15.64:6790/0 conn(0x7f40cc0ab170 legacy=0x7f40cc13d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.027+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 >> v1:172.21.15.64:6789/0 conn(0x7f40cc0af0f0 legacy=0x7f40cc13e030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.028+0000 7f40b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.028+0000 7f40b3fff640 1 monclient: found mon.b 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc13b3a0 con 0x7f40cc11f1e0 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:21.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:21.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 5 monclient: authenticate success, global_id 4618 2024-03-29T11:34:21.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 10 monclient: _renew_subs 2024-03-29T11:34:21.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:21.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40cc13b650 con 0x7f40cc11f1e0 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 10 monclient: _renew_subs 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40cc1e72e0 con 0x7f40cc11f1e0 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.029+0000 7f40d1f3d640 1 librados: init done 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.031+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40c40020b0 con 0x7f40cc11f1e0 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.031+0000 7f40b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.031+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f40c4003c20 con 0x7f40cc11f1e0 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.031+0000 7f40b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.031+0000 7f40b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.031+0000 7f40b3fff640 10 monclient: dump: 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:21.032 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:21.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.033+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f40c4003e80 con 0x7f40cc11f1e0 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.034+0000 7f40c9ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.034+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7f40c4084e70 con 0x7f40cc11f1e0 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.034+0000 7f40b3fff640 10 client.4618.objecter ms_dispatch 0x7f40cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.034+0000 7f40b3fff640 3 client.4618.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.034+0000 7f40b3fff640 3 client.4618.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.034+0000 7f40b3fff640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.035+0000 7f40d1f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.035+0000 7f40d1f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.035+0000 7f40d1f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.035+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac005080 con 0x7f40cc11f1e0 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.035+0000 7f40cb7fe640 10 monclient: get_auth_request con 0x7f40b806dc00 auth_method 0 2024-03-29T11:34:21.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.036+0000 7f40b3fff640 10 client.4618.objecter ms_handle_connect 0x7f40b806dc00 2024-03-29T11:34:21.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.041+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40c4050750 con 0x7f40cc11f1e0 2024-03-29T11:34:21.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.041+0000 7f40b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:21.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.041+0000 7f40b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:21.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.405+0000 7f40d1f3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:21.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.405+0000 7f40d1f3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.405+0000 7f40d1f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.405+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40ac005330 con 0x7f40cc11f1e0 2024-03-29T11:34:21.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.410+0000 7f40b3fff640 1 -- 172.21.15.64:0/1983575427 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7f40c4052680 con 0x7f40cc11f1e0 2024-03-29T11:34:21.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.410+0000 7f40b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:21.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.410+0000 7f40b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:21.408 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:21.408 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:21.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.413+0000 7f40d1f3d640 10 librados: watch_flush enter 2024-03-29T11:34:21.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 10 librados: watch_flush exit 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 10 client.4618.objecter successfully canceled tick 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 >> v1:172.21.15.136:6816/2312274743 conn(0x7f40b806dc00 legacy=0x7f40b80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 10 monclient: shutdown 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.414+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 >> v1:172.21.15.136:6789/0 conn(0x7f40cc11f1e0 legacy=0x7f40cc13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.416+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 shutdown_connections 2024-03-29T11:34:21.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.416+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 >> 172.21.15.64:0/1983575427 conn(0x7f40cc0b35e0 msgr2=0x7f40cc0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:21.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.416+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 shutdown_connections 2024-03-29T11:34:21.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.416+0000 7f40d1f3d640 1 -- 172.21.15.64:0/1983575427 wait complete. 2024-03-29T11:34:21.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:21.417+0000 7f40d1f3d640 1 librados: shutdown 2024-03-29T11:34:21.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:28.238 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:28.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.393+0000 7f2cb9084640 10 monclient: get_monmap_and_config 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.393+0000 7f2cb9084640 10 monclient: build_initial_monmap 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.393+0000 7f2cb9084640 10 monclient: monmap: 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:28.394655+0000 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:28.394655+0000 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:28.391 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.394+0000 7f2cb9084640 1 Processor -- start 2024-03-29T11:34:28.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.394+0000 7f2cb9084640 1 -- start start 2024-03-29T11:34:28.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.394+0000 7f2cb9084640 10 monclient: init 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb4070120 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb40b0420 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb4070500 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4072090 con 0x7f2cb4070500 2024-03-29T11:34:28.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4072370 con 0x7f2cb4070120 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4072650 con 0x7f2cb40b0420 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb9084640 10 monclient(hunting): authenticate will time out at 1804.665528s 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb359e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2cb4070500 0x7f2cb414b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43778/0 (socket says 172.21.15.64:43778) 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.395+0000 7f2cb359e640 1 -- 172.21.15.64:0/2816477441 learned_addr learned my addr 172.21.15.64:0/2816477441 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874542877 0 0) 0x7f2cb4072650 con 0x7f2cb40b0420 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9c003520 con 0x7f2cb40b0420 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50848718 0 0) 0x7f2cb4072090 con 0x7f2cb4070500 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:34:28.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:28.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:28.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:34:28.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb4072650 con 0x7f2cb4070500 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142309937 0 0) 0x7f2c9c003520 con 0x7f2cb40b0420 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 >> v1:172.21.15.136:6789/0 conn(0x7f2cb4070120 legacy=0x7f2cb413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 >> v1:172.21.15.64:6789/0 conn(0x7f2cb4070500 legacy=0x7f2cb414b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 monclient: found mon.noname-b 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:34:28.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb406dca0 con 0x7f2cb40b0420 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb9084640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb9084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ca80015c0 con 0x7f2cb40b0420 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.396+0000 7f2cb1d9b640 10 monclient: dump: 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:28.397 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ca80017a0 con 0x7f2cb40b0420 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb1d9b640 1 -- 172.21.15.64:0/2816477441 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ca80015c0 con 0x7f2cb40b0420 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb1d9b640 10 monclient: dump: 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.398 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb9084640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb9084640 4 set_mon_vals no callback set 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb9084640 10 monclient: shutdown 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb9084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.397+0000 7f2cb9084640 1 -- 172.21.15.64:0/2816477441 >> v1:172.21.15.64:6790/0 conn(0x7f2cb40b0420 legacy=0x7f2cb41405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.398+0000 7f2cb9084640 1 -- 172.21.15.64:0/2816477441 shutdown_connections 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.398+0000 7f2cb9084640 1 -- 172.21.15.64:0/2816477441 >> 172.21.15.64:0/2816477441 conn(0x7f2cb4067500 msgr2=0x7f2cb40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.398+0000 7f2cb9084640 1 -- 172.21.15.64:0/2816477441 shutdown_connections 2024-03-29T11:34:28.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.398+0000 7f2cb9084640 1 -- 172.21.15.64:0/2816477441 wait complete. 2024-03-29T11:34:28.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.399+0000 7f2cb9084640 10 monclient: build_initial_monmap 2024-03-29T11:34:28.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.399+0000 7f2cb9084640 10 monclient: monmap: 2024-03-29T11:34:28.406 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:28.406 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.399+0000 7f2cb9084640 1 librados: starting msgr at 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.399+0000 7f2cb9084640 1 librados: starting objecter 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.400+0000 7f2cb9084640 1 Processor -- start 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.400+0000 7f2cb9084640 1 -- start start 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.400+0000 7f2cb9084640 1 librados: setting wanted keys 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.400+0000 7f2cb9084640 1 librados: calling monclient init 2024-03-29T11:34:28.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.400+0000 7f2cb9084640 10 monclient: init 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.400+0000 7f2cb9084640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb4070120 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb4070500 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb40b0420 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb41387b0 con 0x7f2cb4070120 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40a9eb0 con 0x7f2cb4070500 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb2d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2cb4070120 0x7f2cb41d7130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43996/0 (socket says 172.21.15.64:43996) 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb359e640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2cb40b0420 0x7f2cb41c9d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:57320/0 (socket says 172.21.15.64:57320) 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb2d9d640 1 -- 172.21.15.64:0/2040515648 learned_addr learned my addr 172.21.15.64:0/2040515648 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb41caa10 con 0x7f2cb40b0420 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2cb9084640 10 monclient(hunting): authenticate will time out at 1804.671269s 2024-03-29T11:34:28.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2c9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2cb40b0420 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2c9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2c9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2cb4070120 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.401+0000 7f2c9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2cb4070500 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025829296 0 0) 0x7f2cb40a9eb0 con 0x7f2cb4070500 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): my global_id is 4621 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0004eb0 con 0x7f2cb4070500 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990678835 0 0) 0x7f2cb41caa10 con 0x7f2cb40b0420 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:28.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.402+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb40a9eb0 con 0x7f2cb40b0420 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2568668202 0 0) 0x7f2ca0004eb0 con 0x7f2cb4070500 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb41caa10 con 0x7f2cb4070500 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640394506 0 0) 0x7f2cb41387b0 con 0x7f2cb4070120 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0004eb0 con 0x7f2cb4070120 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ca80015c0 con 0x7f2cb4070500 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:28.410 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1382052002 0 0) 0x7f2cb40a9eb0 con 0x7f2cb40b0420 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb41387b0 con 0x7f2cb40b0420 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2cac001c70 con 0x7f2cb40b0420 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:28.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.403+0000 7f2c9b7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1267234211 0 0) 0x7f2ca0004eb0 con 0x7f2cb4070120 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb40a9eb0 con 0x7f2cb4070120 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ca4001820 con 0x7f2cb4070120 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:28.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3588822664 0 0) 0x7f2cb41caa10 con 0x7f2cb4070500 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 >> v1:172.21.15.64:6790/0 conn(0x7f2cb40b0420 legacy=0x7f2cb41c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 >> v1:172.21.15.64:6789/0 conn(0x7f2cb4070120 legacy=0x7f2cb41d7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 monclient: found mon.b 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb41ca480 con 0x7f2cb4070500 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2c9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 5 monclient: authenticate success, global_id 4621 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 10 monclient: _renew_subs 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb41ca730 con 0x7f2cb4070500 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 10 monclient: _renew_subs 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:28.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb4138b40 con 0x7f2cb4070500 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.404+0000 7f2cb9084640 1 librados: init done 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.406+0000 7f2cb9084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.406+0000 7f2cb9084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.406+0000 7f2cb9084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.406+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb4075f10 con 0x7f2cb4070500 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ca8001ce0 con 0x7f2cb4070500 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2c9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f2ca80038c0 con 0x7f2cb4070500 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2c9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2c9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2c9b7fe640 10 monclient: dump: 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.415 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:28.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.407+0000 7f2cb159a640 4 set_mon_vals no callback set 2024-03-29T11:34:28.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.409+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f2ca8019890 con 0x7f2cb4070500 2024-03-29T11:34:28.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.410+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7f2ca8083710 con 0x7f2cb4070500 2024-03-29T11:34:28.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.410+0000 7f2c9b7fe640 10 client.4621.objecter ms_dispatch 0x7f2cb4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:28.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.410+0000 7f2c9b7fe640 3 client.4621.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:28.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.410+0000 7f2c9b7fe640 3 client.4621.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:28.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.410+0000 7f2cb2d9d640 10 monclient: get_auth_request con 0x7f2ca006e190 auth_method 0 2024-03-29T11:34:28.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.410+0000 7f2c9b7fe640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-29T11:34:28.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.411+0000 7f2c9b7fe640 10 client.4621.objecter ms_handle_connect 0x7f2ca006e190 2024-03-29T11:34:28.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.416+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ca804ee20 con 0x7f2cb4070500 2024-03-29T11:34:28.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.416+0000 7f2c9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:28.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.416+0000 7f2c9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:28.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.783+0000 7f2cb9084640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:28.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.783+0000 7f2cb9084640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:28.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.783+0000 7f2cb9084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:28.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.783+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cb4070f50 con 0x7f2cb4070500 2024-03-29T11:34:28.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.784+0000 7f2c9b7fe640 1 -- 172.21.15.64:0/2040515648 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7f2ca8050f20 con 0x7f2cb4070500 2024-03-29T11:34:28.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.784+0000 7f2c9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:28.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.784+0000 7f2c9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:28.782 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:28.782 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.788+0000 7f2cb9084640 10 librados: watch_flush enter 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 10 librados: watch_flush exit 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 10 client.4621.objecter successfully canceled tick 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 >> v1:172.21.15.136:6816/2312274743 conn(0x7f2ca006e190 legacy=0x7f2ca0070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 10 monclient: shutdown 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 >> v1:172.21.15.136:6789/0 conn(0x7f2cb4070500 legacy=0x7f2cb41c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 shutdown_connections 2024-03-29T11:34:28.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 >> 172.21.15.64:0/2040515648 conn(0x7f2cb4067500 msgr2=0x7f2cb406d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:28.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 shutdown_connections 2024-03-29T11:34:28.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 -- 172.21.15.64:0/2040515648 wait complete. 2024-03-29T11:34:28.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:28.789+0000 7f2cb9084640 1 librados: shutdown 2024-03-29T11:34:28.804 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:35.611 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.770+0000 7fa18ea4a640 10 monclient: get_monmap_and_config 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.770+0000 7fa18ea4a640 10 monclient: build_initial_monmap 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.770+0000 7fa18ea4a640 10 monclient: monmap: 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:35.771927+0000 2024-03-29T11:34:35.768 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:35.771927+0000 2024-03-29T11:34:35.769 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:35.769 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.769 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:35.769 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:35.769 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:35.769 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.773+0000 7fa18ea4a640 1 Processor -- start 2024-03-29T11:34:35.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.775+0000 7fa18ea4a640 1 -- start start 2024-03-29T11:34:35.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.775+0000 7fa18ea4a640 10 monclient: init 2024-03-29T11:34:35.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.776+0000 7fa18ea4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:35.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.776+0000 7fa18ea4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:35.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.776+0000 7fa18ea4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1881457f0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:35.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.776+0000 7fa18ea4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa18813f650 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:35.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.779+0000 7fa18ea4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1880a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa18ea4a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1880aaef0 con 0x7fa18813f650 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa18ea4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa188145bd0 con 0x7fa1881457f0 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa18ea4a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa188145eb0 con 0x7fa1880a9d70 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa18ea4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa18ea4a640 10 monclient(hunting): authenticate will time out at 1812.050403s 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa1877fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa18813f650 0x7fa1880b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55847/0 (socket says 172.21.15.64:55847) 2024-03-29T11:34:35.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.780+0000 7fa1877fe640 1 -- 172.21.15.64:0/1932668795 learned_addr learned my addr 172.21.15.64:0/1932668795 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342539465 0 0) 0x7fa1880aaef0 con 0x7fa18813f650 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa170003520 con 0x7fa18813f650 2024-03-29T11:34:35.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465919556 0 0) 0x7fa188145eb0 con 0x7fa1880a9d70 2024-03-29T11:34:35.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:34:35.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa170003840 con 0x7fa1880a9d70 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716095466 0 0) 0x7fa170003520 con 0x7fa18813f650 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 >> v1:172.21.15.64:6790/0 conn(0x7fa1880a9d70 legacy=0x7fa18814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 >> v1:172.21.15.136:6789/0 conn(0x7fa1881457f0 legacy=0x7fa18813ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 monclient: found mon.noname-a 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa188146340 con 0x7fa18813f650 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa1800019f0 con 0x7fa18813f650 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.783+0000 7fa186ffd640 10 monclient: dump: 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa180001bd0 con 0x7fa18813f650 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa186ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa186ffd640 1 -- 172.21.15.64:0/1932668795 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa1800019f0 con 0x7fa18813f650 2024-03-29T11:34:35.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa186ffd640 10 monclient: dump: 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa18ea4a640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa18ea4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:35.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.784+0000 7fa18ea4a640 4 set_mon_vals no callback set 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.785+0000 7fa18ea4a640 10 monclient: shutdown 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.785+0000 7fa18ea4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.785+0000 7fa18ea4a640 1 -- 172.21.15.64:0/1932668795 >> v1:172.21.15.64:6789/0 conn(0x7fa18813f650 legacy=0x7fa1880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.785+0000 7fa18ea4a640 1 -- 172.21.15.64:0/1932668795 shutdown_connections 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.785+0000 7fa18ea4a640 1 -- 172.21.15.64:0/1932668795 >> 172.21.15.64:0/1932668795 conn(0x7fa188066a00 msgr2=0x7fa188066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.787+0000 7fa18ea4a640 1 -- 172.21.15.64:0/1932668795 shutdown_connections 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.787+0000 7fa18ea4a640 1 -- 172.21.15.64:0/1932668795 wait complete. 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.788+0000 7fa18ea4a640 10 monclient: build_initial_monmap 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.788+0000 7fa18ea4a640 10 monclient: monmap: 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:35.789 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:35.790 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.790 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:35.790 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:35.790 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:35.790 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.788+0000 7fa18ea4a640 1 librados: starting msgr at 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.789+0000 7fa18ea4a640 1 librados: starting objecter 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.789+0000 7fa18ea4a640 1 Processor -- start 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.789+0000 7fa18ea4a640 1 -- start start 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.789+0000 7fa18ea4a640 1 librados: setting wanted keys 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.789+0000 7fa18ea4a640 1 librados: calling monclient init 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.789+0000 7fa18ea4a640 10 monclient: init 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa1880a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:35.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa18813f650 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:35.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1881457f0 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:35.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1881d71a0 con 0x7fa1880a9d70 2024-03-29T11:34:35.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18806eab0 con 0x7fa18813f650 2024-03-29T11:34:35.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1881d7440 con 0x7fa1881457f0 2024-03-29T11:34:35.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:35.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.791+0000 7fa18ea4a640 10 monclient(hunting): authenticate will time out at 1812.061071s 2024-03-29T11:34:35.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.797+0000 7fa1877fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa18813f650 0x7fa188139a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:54051/0 (socket says 172.21.15.64:54051) 2024-03-29T11:34:35.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.797+0000 7fa1877fe640 1 -- 172.21.15.64:0/2214165124 learned_addr learned my addr 172.21.15.64:0/2214165124 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:35.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 client.?.objecter ms_handle_connect 0x7fa18813f650 2024-03-29T11:34:35.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470203585 0 0) 0x7fa18806eab0 con 0x7fa18813f650 2024-03-29T11:34:35.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): my global_id is 4624 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa174004eb0 con 0x7fa18813f650 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 client.?.objecter ms_handle_connect 0x7fa1881457f0 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174257189 0 0) 0x7fa1881d7440 con 0x7fa1881457f0 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.798+0000 7fa184ff9640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:34:35.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa18806eab0 con 0x7fa1881457f0 2024-03-29T11:34:35.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 client.?.objecter ms_handle_connect 0x7fa1880a9d70 2024-03-29T11:34:35.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231604804 0 0) 0x7fa174004eb0 con 0x7fa18813f650 2024-03-29T11:34:35.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1881d7440 con 0x7fa18813f650 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2922666894 0 0) 0x7fa18806eab0 con 0x7fa1881457f0 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa174004eb0 con 0x7fa1881457f0 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa1800019f0 con 0x7fa18813f650 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:35.799 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa1780018b0 con 0x7fa1881457f0 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.799+0000 7fa184ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:35.800 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1597191172 0 0) 0x7fa1881d7440 con 0x7fa18813f650 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 >> v1:172.21.15.64:6790/0 conn(0x7fa1881457f0 legacy=0x7fa18813a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 >> v1:172.21.15.64:6789/0 conn(0x7fa1880a9d70 legacy=0x7fa1881d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 monclient: found mon.b 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa188142410 con 0x7fa18813f650 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa180002060 con 0x7fa18813f650 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:35.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7fa180005310 con 0x7fa18813f650 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa184ff9640 10 monclient: dump: 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa185ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:35.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa18ea4a640 5 monclient: authenticate success, global_id 4624 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa18ea4a640 10 monclient: _renew_subs 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa18ea4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa18ea4a640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1881426e0 con 0x7fa18813f650 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa18ea4a640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.800+0000 7fa18ea4a640 10 monclient: _renew_subs 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.801+0000 7fa18ea4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.801+0000 7fa18ea4a640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1881ded00 con 0x7fa18813f650 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.801+0000 7fa18ea4a640 1 librados: init done 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.803+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7fa18001b2e0 con 0x7fa18813f650 2024-03-29T11:34:35.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.804+0000 7fa187fff640 10 monclient: get_auth_request con 0x7fa17406df10 auth_method 0 2024-03-29T11:34:35.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.804+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7fa180085160 con 0x7fa18813f650 2024-03-29T11:34:35.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.804+0000 7fa184ff9640 10 client.4624.objecter ms_dispatch 0x7fa188000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:35.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.804+0000 7fa184ff9640 3 client.4624.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:35.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.804+0000 7fa184ff9640 3 client.4624.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:35.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.804+0000 7fa184ff9640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-29T11:34:35.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.805+0000 7fa16e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.805+0000 7fa16e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.805+0000 7fa16e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:35.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.805+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa188075f10 con 0x7fa18813f650 2024-03-29T11:34:35.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.807+0000 7fa184ff9640 10 client.4624.objecter ms_handle_connect 0x7fa17406df10 2024-03-29T11:34:35.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.815+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa180050a40 con 0x7fa18813f650 2024-03-29T11:34:35.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.815+0000 7fa184ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:35.815+0000 7fa184ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:36.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.149+0000 7fa16e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:36.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.149+0000 7fa16e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:36.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.149+0000 7fa16e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:36.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.149+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1880b3680 con 0x7fa18813f650 2024-03-29T11:34:36.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.150+0000 7fa184ff9640 1 -- 172.21.15.64:0/2214165124 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7fa180052970 con 0x7fa18813f650 2024-03-29T11:34:36.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.150+0000 7fa184ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:36.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.150+0000 7fa184ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:36.150 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:36.150 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 10 librados: watch_flush enter 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 10 librados: watch_flush exit 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 10 client.4624.objecter successfully canceled tick 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 >> v1:172.21.15.136:6816/2312274743 conn(0x7fa17406df10 legacy=0x7fa1740703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 10 monclient: shutdown 2024-03-29T11:34:36.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:36.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 >> v1:172.21.15.136:6789/0 conn(0x7fa18813f650 legacy=0x7fa188139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:36.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 shutdown_connections 2024-03-29T11:34:36.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.156+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 >> 172.21.15.64:0/2214165124 conn(0x7fa188066a00 msgr2=0x7fa18813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:36.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.158+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 shutdown_connections 2024-03-29T11:34:36.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.158+0000 7fa16e7fc640 1 -- 172.21.15.64:0/2214165124 wait complete. 2024-03-29T11:34:36.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:36.160+0000 7fa16e7fc640 1 librados: shutdown 2024-03-29T11:34:36.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:42.983 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:43.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.145+0000 7ffb31055640 10 monclient: get_monmap_and_config 2024-03-29T11:34:43.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.145+0000 7ffb31055640 10 monclient: build_initial_monmap 2024-03-29T11:34:43.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.145+0000 7ffb31055640 10 monclient: monmap: 2024-03-29T11:34:43.143 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:43.143 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:43.146030+0000 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:43.146030+0000 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.145+0000 7ffb31055640 1 Processor -- start 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 1 -- start start 2024-03-29T11:34:43.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient: init 2024-03-29T11:34:43.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:43.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:43.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient(hunting): picked mon.noname-c con 0x7ffb2c0ae250 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:43.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient(hunting): picked mon.noname-b con 0x7ffb2c0a9720 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:43.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient(hunting): picked mon.noname-a con 0x7ffb2c14bba0 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:43.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0a9b80 con 0x7ffb2c14bba0 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0aa7b0 con 0x7ffb2c0ae250 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0afbf0 con 0x7ffb2c0a9720 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.146+0000 7ffb31055640 10 monclient(hunting): authenticate will time out at 1819.416816s 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb2b59e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ffb2c14bba0 0x7ffb2c14df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35169/0 (socket says 172.21.15.64:35169) 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb2b59e640 1 -- 172.21.15.64:0/2284675128 learned_addr learned my addr 172.21.15.64:0/2284675128 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962232537 0 0) 0x7ffb2c0a9b80 con 0x7ffb2c14bba0 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:34:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb14003520 con 0x7ffb2c14bba0 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986811031 0 0) 0x7ffb14003520 con 0x7ffb2c14bba0 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.150+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 >> v1:172.21.15.64:6790/0 conn(0x7ffb2c0a9720 legacy=0x7ffb2c13f7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.153+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 >> v1:172.21.15.136:6789/0 conn(0x7ffb2c0ae250 legacy=0x7ffb2c142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.153+0000 7ffb29d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.153+0000 7ffb29d9b640 1 monclient: found mon.noname-a 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c143280 con 0x7ffb2c14bba0 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: _finish_auth 0 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb31055640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:34:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb31055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ffb24001a10 con 0x7ffb2c14bba0 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: dump: 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:43.156 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb31055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24001c30 con 0x7ffb2c14bba0 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb31055640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.154+0000 7ffb31055640 4 set_mon_vals no callback set 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb29d9b640 1 -- 172.21.15.64:0/2284675128 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ffb24001fc0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb29d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb29d9b640 10 monclient: dump: 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.157 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 10 monclient: shutdown 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 1 -- 172.21.15.64:0/2284675128 >> v1:172.21.15.64:6789/0 conn(0x7ffb2c14bba0 legacy=0x7ffb2c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 1 -- 172.21.15.64:0/2284675128 shutdown_connections 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 1 -- 172.21.15.64:0/2284675128 >> 172.21.15.64:0/2284675128 conn(0x7ffb2c0b35e0 msgr2=0x7ffb2c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 1 -- 172.21.15.64:0/2284675128 shutdown_connections 2024-03-29T11:34:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.155+0000 7ffb31055640 1 -- 172.21.15.64:0/2284675128 wait complete. 2024-03-29T11:34:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.156+0000 7ffb31055640 10 monclient: build_initial_monmap 2024-03-29T11:34:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.156+0000 7ffb31055640 10 monclient: monmap: 2024-03-29T11:34:43.160 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 1 librados: starting msgr at 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 1 librados: starting objecter 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 1 Processor -- start 2024-03-29T11:34:43.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 1 -- start start 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 1 librados: setting wanted keys 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 1 librados: calling monclient init 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.157+0000 7ffb31055640 10 monclient: init 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient(hunting): picked mon.noname-a con 0x7ffb2c0a9720 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient(hunting): picked mon.noname-c con 0x7ffb2c0ae250 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient(hunting): picked mon.noname-b con 0x7ffb2c14bba0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c06ff60 con 0x7ffb2c0a9720 2024-03-29T11:34:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0711d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c071440 con 0x7ffb2c0ae250 2024-03-29T11:34:43.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:43.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.158+0000 7ffb31055640 10 monclient(hunting): authenticate will time out at 1819.428649s 2024-03-29T11:34:43.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.165+0000 7ffb2b59e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffb2c14bba0 0x7ffb2c1ee8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:41495/0 (socket says 172.21.15.64:41495) 2024-03-29T11:34:43.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.165+0000 7ffb2b59e640 1 -- 172.21.15.64:0/4062524176 learned_addr learned my addr 172.21.15.64:0/4062524176 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:43.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 client.?.objecter ms_handle_connect 0x7ffb2c14bba0 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318226037 0 0) 0x7ffb2c0711d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 monclient(hunting): my global_id is 4627 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18004eb0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 client.?.objecter ms_handle_connect 0x7ffb2c0a9720 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.166+0000 7ffb137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901357814 0 0) 0x7ffb18004eb0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb2c0711d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ffb240018d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): dump: 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:43.172 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741884320 0 0) 0x7ffb2c06ff60 con 0x7ffb2c0a9720 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.167+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18004eb0 con 0x7ffb2c0a9720 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754736741 0 0) 0x7ffb2c0711d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 >> v1:172.21.15.64:6790/0 conn(0x7ffb2c0ae250 legacy=0x7ffb2c071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 >> v1:172.21.15.64:6789/0 conn(0x7ffb2c0a9720 legacy=0x7ffb2c06f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 1 monclient: found mon.b 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:43.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c071720 con 0x7ffb2c14bba0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 5 monclient: authenticate success, global_id 4627 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 10 monclient: _renew_subs 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2c06fbf0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 10 monclient: _renew_subs 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c072490 con 0x7ffb2c14bba0 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.168+0000 7ffb31055640 1 librados: init done 2024-03-29T11:34:43.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24004e80 con 0x7ffb2c14bba0 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7ffb240052d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb137fe640 10 monclient: dump: 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:43.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:43.176 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:43.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.169+0000 7ffb2959a640 4 set_mon_vals no callback set 2024-03-29T11:34:43.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.170+0000 7ffb31055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:43.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.170+0000 7ffb31055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:43.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.170+0000 7ffb31055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:43.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.170+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c005080 con 0x7ffb2c14bba0 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.174+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7ffb24005430 con 0x7ffb2c14bba0 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.174+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7ffb24084ec0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.175+0000 7ffb2ad9d640 10 monclient: get_auth_request con 0x7ffb1806dd70 auth_method 0 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.175+0000 7ffb137fe640 10 client.4627.objecter ms_dispatch 0x7ffb2c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.176+0000 7ffb137fe640 3 client.4627.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.176+0000 7ffb137fe640 3 client.4627.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:43.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.176+0000 7ffb137fe640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-29T11:34:43.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.176+0000 7ffb137fe640 10 client.4627.objecter ms_handle_connect 0x7ffb1806dd70 2024-03-29T11:34:43.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.180+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb240507a0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.180+0000 7ffb137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:43.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.180+0000 7ffb137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:43.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.516+0000 7ffb31055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:43.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.516+0000 7ffb31055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:43.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.516+0000 7ffb31055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T11:34:43.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.516+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb0c005330 con 0x7ffb2c14bba0 2024-03-29T11:34:43.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.517+0000 7ffb137fe640 1 -- 172.21.15.64:0/4062524176 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7ffb240526d0 con 0x7ffb2c14bba0 2024-03-29T11:34:43.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.517+0000 7ffb137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:43.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.517+0000 7ffb137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:43.515 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:43.515 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 10 librados: watch_flush enter 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 10 librados: watch_flush exit 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 10 client.4627.objecter successfully canceled tick 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 >> v1:172.21.15.136:6816/2312274743 conn(0x7ffb1806dd70 legacy=0x7ffb18070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 10 monclient: shutdown 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 >> v1:172.21.15.136:6789/0 conn(0x7ffb2c14bba0 legacy=0x7ffb2c1ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 shutdown_connections 2024-03-29T11:34:43.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.521+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 >> 172.21.15.64:0/4062524176 conn(0x7ffb2c0b35e0 msgr2=0x7ffb2c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:43.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.522+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 shutdown_connections 2024-03-29T11:34:43.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.522+0000 7ffb31055640 1 -- 172.21.15.64:0/4062524176 wait complete. 2024-03-29T11:34:43.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:43.522+0000 7ffb31055640 1 librados: shutdown 2024-03-29T11:34:43.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:50.345 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:50.490 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.493+0000 7f4adc9a8640 10 monclient: get_monmap_and_config 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.493+0000 7f4adc9a8640 10 monclient: build_initial_monmap 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.493+0000 7f4adc9a8640 10 monclient: monmap: 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:50.494150+0000 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:50.494150+0000 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:50.491 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:50.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.494+0000 7f4adc9a8640 1 Processor -- start 2024-03-29T11:34:50.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.494+0000 7f4adc9a8640 1 -- start start 2024-03-29T11:34:50.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.494+0000 7f4adc9a8640 10 monclient: init 2024-03-29T11:34:50.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:50.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:50.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4ad81422b0 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:50.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4ad8142690 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:50.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4ad80b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:50.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad8142c80 con 0x7f4ad8142690 2024-03-29T11:34:50.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad80aae40 con 0x7f4ad80b17d0 2024-03-29T11:34:50.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad813fbd0 con 0x7f4ad81422b0 2024-03-29T11:34:50.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:50.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.497+0000 7f4adc9a8640 10 monclient(hunting): authenticate will time out at 1826.767044s 2024-03-29T11:34:50.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.498+0000 7f4ad5d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4ad8142690 0x7f4ad80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40082/0 (socket says 172.21.15.64:40082) 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.498+0000 7f4ad5d9b640 1 -- 172.21.15.64:0/2701111203 learned_addr learned my addr 172.21.15.64:0/2701111203 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745385561 0 0) 0x7f4ad8142c80 con 0x7f4ad8142690 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac0003520 con 0x7f4ad8142690 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1669677334 0 0) 0x7f4ac0003520 con 0x7f4ad8142690 2024-03-29T11:34:50.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 >> v1:172.21.15.64:6790/0 conn(0x7f4ad81422b0 legacy=0x7f4ad813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:50.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.499+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 >> v1:172.21.15.136:6789/0 conn(0x7f4ad80b17d0 legacy=0x7f4ad80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 1 monclient: found mon.noname-a 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8151940 con 0x7f4ad8142690 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: _finish_auth 0 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4acc001690 con 0x7f4ad8142690 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:50.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: dump: 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4acc0018b0 con 0x7f4ad8142690 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.500+0000 7f4ad559a640 1 -- 172.21.15.64:0/2701111203 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4acc001c80 con 0x7f4ad8142690 2024-03-29T11:34:50.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.501+0000 7f4ad559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.501+0000 7f4ad559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.501+0000 7f4ad559a640 10 monclient: dump: 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:50.502 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.501+0000 7f4adc9a8640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.501+0000 7f4adc9a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.501+0000 7f4adc9a8640 4 set_mon_vals no callback set 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.502+0000 7f4adc9a8640 10 monclient: shutdown 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.502+0000 7f4adc9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.502+0000 7f4adc9a8640 1 -- 172.21.15.64:0/2701111203 >> v1:172.21.15.64:6789/0 conn(0x7f4ad8142690 legacy=0x7f4ad80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.503+0000 7f4adc9a8640 1 -- 172.21.15.64:0/2701111203 shutdown_connections 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.503+0000 7f4adc9a8640 1 -- 172.21.15.64:0/2701111203 >> 172.21.15.64:0/2701111203 conn(0x7f4ad8066a00 msgr2=0x7f4ad8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.503+0000 7f4adc9a8640 1 -- 172.21.15.64:0/2701111203 shutdown_connections 2024-03-29T11:34:50.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.503+0000 7f4adc9a8640 1 -- 172.21.15.64:0/2701111203 wait complete. 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.504+0000 7f4adc9a8640 10 monclient: build_initial_monmap 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.504+0000 7f4adc9a8640 10 monclient: monmap: 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 1 librados: starting msgr at 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 1 librados: starting objecter 2024-03-29T11:34:50.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 1 Processor -- start 2024-03-29T11:34:50.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 1 -- start start 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 1 librados: setting wanted keys 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 1 librados: calling monclient init 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.505+0000 7f4adc9a8640 10 monclient: init 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4ad80b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4ad81422b0 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4ad81dbfb0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad806f0d0 con 0x7f4ad81422b0 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad806f3b0 con 0x7f4ad81dbfb0 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad81326b0 con 0x7f4ad80b17d0 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:50.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.506+0000 7f4adc9a8640 10 monclient(hunting): authenticate will time out at 1826.776240s 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.509+0000 7f4ad5d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4ad81422b0 0x7f4ad81db8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40860/0 (socket says 172.21.15.64:40860) 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.509+0000 7f4ad5d9b640 1 -- 172.21.15.64:0/3303241198 learned_addr learned my addr 172.21.15.64:0/3303241198 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ad81422b0 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ad81dbfb0 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978094575 0 0) 0x7f4ad806f0d0 con 0x7f4ad81422b0 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:34:50.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.510+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab8004eb0 con 0x7f4ad81422b0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ad80b17d0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 732037724 0 0) 0x7f4ab8004eb0 con 0x7f4ad81422b0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ad806f0d0 con 0x7f4ad81422b0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445982611 0 0) 0x7f4ad806f3b0 con 0x7f4ad81dbfb0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 monclient(hunting): my global_id is 4630 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab8004eb0 con 0x7f4ad81dbfb0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175739992 0 0) 0x7f4ad81326b0 con 0x7f4ad80b17d0 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.511+0000 7f4ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:50.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad806f3b0 con 0x7f4ad80b17d0 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4acc001550 con 0x7f4ad81422b0 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:50.517 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710510999 0 0) 0x7f4ab8004eb0 con 0x7f4ad81dbfb0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ad81326b0 con 0x7f4ad81dbfb0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3453309871 0 0) 0x7f4ad806f3b0 con 0x7f4ad80b17d0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.512+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ab8004eb0 con 0x7f4ad80b17d0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3903509243 0 0) 0x7f4ad806f0d0 con 0x7f4ad81422b0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 >> v1:172.21.15.64:6790/0 conn(0x7f4ad80b17d0 legacy=0x7f4ad8131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 >> v1:172.21.15.136:6789/0 conn(0x7f4ad81dbfb0 legacy=0x7f4ad812ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 1 monclient: found mon.a 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad813bba0 con 0x7f4ad81422b0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:50.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 10 monclient: _renew_subs 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad813bd90 con 0x7f4ad81422b0 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 10 monclient: _renew_subs 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad812f4a0 con 0x7f4ad81422b0 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.513+0000 7f4adc9a8640 1 librados: init done 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4adc9a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4adc9a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4adc9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad8075f10 con 0x7f4ad81422b0 2024-03-29T11:34:50.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4acc0016b0 con 0x7f4ad81422b0 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4ac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f4acc004d90 con 0x7f4ad81422b0 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4ac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.515+0000 7f4ac6ffd640 10 monclient: dump: 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:50.520 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.516+0000 7f4ad4d99640 4 set_mon_vals no callback set 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.517+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f4acc01ad60 con 0x7f4ad81422b0 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ad659c640 10 monclient: get_auth_request con 0x7f4ab806e320 auth_method 0 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7f4acc084b40 con 0x7f4ad81422b0 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ac6ffd640 10 client.4596.objecter ms_dispatch 0x7f4ad8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ac6ffd640 3 client.4596.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ac6ffd640 3 client.4596.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ac6ffd640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:34:50.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.518+0000 7f4ac6ffd640 10 client.4596.objecter ms_handle_connect 0x7f4ab806e320 2024-03-29T11:34:50.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.525+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4acc050420 con 0x7f4ad81422b0 2024-03-29T11:34:50.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.525+0000 7f4ac6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:50.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.525+0000 7f4ac6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:50.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.863+0000 7f4adc9a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:50.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.863+0000 7f4adc9a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:50.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.863+0000 7f4adc9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:50.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.863+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ad80b2270 con 0x7f4ad81422b0 2024-03-29T11:34:50.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.864+0000 7f4ac6ffd640 1 -- 172.21.15.64:0/3303241198 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7f4acc052350 con 0x7f4ad81422b0 2024-03-29T11:34:50.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.864+0000 7f4ac6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:50.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.864+0000 7f4ac6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:50.862 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:50.862 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:50.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.867+0000 7f4adc9a8640 10 librados: watch_flush enter 2024-03-29T11:34:50.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.868+0000 7f4adc9a8640 10 librados: watch_flush exit 2024-03-29T11:34:50.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.868+0000 7f4adc9a8640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:34:50.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.868+0000 7f4adc9a8640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 >> v1:172.21.15.136:6816/2312274743 conn(0x7f4ab806e320 legacy=0x7f4ab80707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 10 monclient: shutdown 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 >> v1:172.21.15.64:6789/0 conn(0x7f4ad81422b0 legacy=0x7f4ad81db8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 shutdown_connections 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 >> 172.21.15.64:0/3303241198 conn(0x7f4ad8066a00 msgr2=0x7f4ad80b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 shutdown_connections 2024-03-29T11:34:50.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.869+0000 7f4adc9a8640 1 -- 172.21.15.64:0/3303241198 wait complete. 2024-03-29T11:34:50.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:50.870+0000 7f4adc9a8640 1 librados: shutdown 2024-03-29T11:34:50.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:57.691 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:57.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.840+0000 7f99b4332640 10 monclient: get_monmap_and_config 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.840+0000 7f99b4332640 10 monclient: build_initial_monmap 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.840+0000 7f99b4332640 10 monclient: monmap: 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:34:57.841599+0000 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:34:57.841599+0000 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T11:34:57.839 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.841+0000 7f99b4332640 1 Processor -- start 2024-03-29T11:34:57.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.842+0000 7f99b4332640 1 -- start start 2024-03-29T11:34:57.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.842+0000 7f99b4332640 10 monclient: init 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient(hunting): picked mon.noname-b con 0x7f99ac0b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient(hunting): picked mon.noname-c con 0x7f99ac071eb0 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient(hunting): picked mon.noname-a con 0x7f99ac143d10 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac06ef80 con 0x7f99ac143d10 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac06f260 con 0x7f99ac071eb0 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac0b40c0 con 0x7f99ac0b1f60 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b4332640 10 monclient(hunting): authenticate will time out at 1834.113060s 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b18cd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f99ac071eb0 0x7f99ac146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:59914/0 (socket says 172.21.15.64:59914) 2024-03-29T11:34:57.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b28cf640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f99ac143d10 0x7f99ac146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42990/0 (socket says 172.21.15.64:42990) 2024-03-29T11:34:57.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.843+0000 7f99b18cd640 1 -- 172.21.15.64:0/1700774161 learned_addr learned my addr 172.21.15.64:0/1700774161 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:57.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426093024 0 0) 0x7f99ac06f260 con 0x7f99ac071eb0 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): my global_id is 4633 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9998003520 con 0x7f99ac071eb0 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413163961 0 0) 0x7f99ac0b40c0 con 0x7f99ac0b1f60 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ac06f260 con 0x7f99ac0b1f60 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162790497 0 0) 0x7f99ac06ef80 con 0x7f99ac143d10 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ac0b40c0 con 0x7f99ac143d10 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2462005072 0 0) 0x7f99ac06f260 con 0x7f99ac0b1f60 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 >> v1:172.21.15.136:6789/0 conn(0x7f99ac071eb0 legacy=0x7f99ac146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 >> v1:172.21.15.64:6789/0 conn(0x7f99ac143d10 legacy=0x7f99ac146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 monclient: found mon.noname-b 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac155d70 con 0x7f99ac0b1f60 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: _finish_auth 0 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:57.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b4332640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b4332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f999c0015a0 con 0x7f99ac0b1f60 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b10cc640 10 monclient: dump: 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:57.846 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.844+0000 7f99b4332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f999c0017c0 con 0x7f99ac0b1f60 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b10cc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b4332640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b4332640 4 set_mon_vals no callback set 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b10cc640 1 -- 172.21.15.64:0/1700774161 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f999c001b50 con 0x7f99ac0b1f60 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b10cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b10cc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.845+0000 7f99b10cc640 10 monclient: dump: 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.847 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 10 monclient: shutdown 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 1 -- 172.21.15.64:0/1700774161 >> v1:172.21.15.64:6790/0 conn(0x7f99ac0b1f60 legacy=0x7f99ac143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 1 -- 172.21.15.64:0/1700774161 shutdown_connections 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 1 -- 172.21.15.64:0/1700774161 >> 172.21.15.64:0/1700774161 conn(0x7f99ac066a00 msgr2=0x7f99ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 1 -- 172.21.15.64:0/1700774161 shutdown_connections 2024-03-29T11:34:57.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.846+0000 7f99b4332640 1 -- 172.21.15.64:0/1700774161 wait complete. 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 10 monclient: build_initial_monmap 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 10 monclient: monmap: 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.855 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 1 librados: starting msgr at 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 1 librados: starting objecter 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 1 Processor -- start 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 1 -- start start 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.847+0000 7f99b4332640 1 librados: setting wanted keys 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.848+0000 7f99b4332640 1 librados: calling monclient init 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.848+0000 7f99b4332640 10 monclient: init 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.848+0000 7f99b4332640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.848+0000 7f99b4332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.848+0000 7f99b4332640 10 monclient(hunting): picked mon.noname-b con 0x7f99ac0b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T11:34:57.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.848+0000 7f99b4332640 10 monclient(hunting): picked mon.noname-c con 0x7f99ac143d10 addr v1:172.21.15.64:6790/0 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b20ce640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f99ac0b1f60 0x7f99ac13f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.64:33494/0 (socket says 172.21.15.64:33494) 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b4332640 10 monclient(hunting): picked mon.noname-a con 0x7f99ac1547f0 addr v1:172.21.15.64:6789/0 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b20ce640 1 -- 172.21.15.64:0/697755687 learned_addr learned my addr 172.21.15.64:0/697755687 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac149970 con 0x7f99ac1547f0 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac13fba0 con 0x7f99ac0b1f60 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac14ad30 con 0x7f99ac143d10 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b4332640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99b4332640 10 monclient(hunting): authenticate will time out at 1834.119072s 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 client.?.objecter ms_handle_connect 0x7f99ac0b1f60 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 client.?.objecter ms_handle_connect 0x7f99ac143d10 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 client.?.objecter ms_handle_connect 0x7f99ac1547f0 2024-03-29T11:34:57.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366467906 0 0) 0x7f99ac14ad30 con 0x7f99ac143d10 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.849+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9990004eb0 con 0x7f99ac143d10 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413744748 0 0) 0x7f99ac149970 con 0x7f99ac1547f0 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:34:57.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ac14ad30 con 0x7f99ac1547f0 2024-03-29T11:34:57.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2695046052 0 0) 0x7f9990004eb0 con 0x7f99ac143d10 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99ac149970 con 0x7f99ac143d10 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f99a8001880 con 0x7f99ac143d10 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:57.863 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460528823 0 0) 0x7f99ac14ad30 con 0x7f99ac1547f0 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9990004eb0 con 0x7f99ac1547f0 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f99a4001a70 con 0x7f99ac1547f0 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.850+0000 7f99a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:57.864 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3302488763 0 0) 0x7f9990004eb0 con 0x7f99ac1547f0 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 >> v1:172.21.15.64:6790/0 conn(0x7f99ac143d10 legacy=0x7f99ac13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 >> v1:172.21.15.136:6789/0 conn(0x7f99ac0b1f60 legacy=0x7f99ac13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 1 monclient: found mon.a 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac14b010 con 0x7f99ac1547f0 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:57.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 10 monclient: _renew_subs 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac1490d0 con 0x7f99ac1547f0 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 10 monclient: _renew_subs 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac149670 con 0x7f99ac1547f0 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b4332640 1 librados: init done 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99a4001f60 con 0x7f99ac1547f0 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.851+0000 7f99b08cb640 4 set_mon_vals no callback set 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.852+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2408054192 0 0) 0x7f99a4004d00 con 0x7f99ac1547f0 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.852+0000 7f99a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.852+0000 7f99a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:57.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.852+0000 7f99a2ffd640 10 monclient: dump: 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:fsid 4e4fa36b-6805-4310-ad24-7c1ad3ae3959 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T11:23:21.724706+0000 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.854+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2708408607 0 0) 0x7f99a4004ee0 con 0x7f99ac1547f0 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.854+0000 7f99b4332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.854+0000 7f99b4332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.854+0000 7f99b4332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:57.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.854+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994005080 con 0x7f99ac1547f0 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1765128778 0 0) 0x7f99a4084ca0 con 0x7f99ac1547f0 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99a2ffd640 10 client.4602.objecter ms_dispatch 0x7f99ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99a2ffd640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99a2ffd640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99a2ffd640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99b20ce640 10 monclient: get_auth_request con 0x7f999006deb0 auth_method 0 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.855+0000 7f99a2ffd640 10 client.4602.objecter ms_handle_connect 0x7f999006deb0 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.864+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99a4050580 con 0x7f99ac1547f0 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.865+0000 7f99a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:57.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:57.865+0000 7f99a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.241+0000 7f99b4332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.241+0000 7f99b4332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.241+0000 7f99b4332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T11:34:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.241+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9994005330 con 0x7f99ac1547f0 2024-03-29T11:34:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.243+0000 7f99a2ffd640 1 -- 172.21.15.64:0/697755687 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3528229761) 0x7f99a40524b0 con 0x7f99ac1547f0 2024-03-29T11:34:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.243+0000 7f99a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.243+0000 7f99a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:58.241 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T11:34:58.241 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"4e4fa36b-6805-4310-ad24-7c1ad3ae3959","created":"2024-03-29T11:23:59.775785+0000","modified":"2024-03-29T11:34:04.697101+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:04.697101+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f4f1a9b9-aa6f-45a3-96d7-27c45d224ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"41cd56e7-b77b-465b-be8e-86e47bd7e82e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e2d90e3-43b4-415d-ab63-6ca41b99d92c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"aaade9b8-7f3c-404e-98c9-c195e943b41a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6d0eda31-ed06-482c-8ff6-a9ed9f1f148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b962f4-e6cb-4a70-a62f-13ac7700c10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aba450cc-a0a8-4404-95c9-a4b048cb684f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"406b222b-5971-4483-bc6a-88ac425573d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 10 librados: watch_flush enter 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 10 librados: watch_flush exit 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 >> v1:172.21.15.136:6816/2312274743 conn(0x7f999006deb0 legacy=0x7f9990070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 10 monclient: shutdown 2024-03-29T11:34:58.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:58.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 >> v1:172.21.15.64:6789/0 conn(0x7f99ac1547f0 legacy=0x7f99ac13b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:58.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 shutdown_connections 2024-03-29T11:34:58.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 >> 172.21.15.64:0/697755687 conn(0x7f99ac066a00 msgr2=0x7f99ac0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:58.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 shutdown_connections 2024-03-29T11:34:58.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 -- 172.21.15.64:0/697755687 wait complete. 2024-03-29T11:34:58.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T11:34:58.246+0000 7f99b4332640 1 librados: shutdown 2024-03-29T11:34:58.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:58.260 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:34:58.262 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:34:58.263 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:34:58.263 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.297 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:34:58.297 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:34:58.297 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.369 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:34:58.370 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:34:58.370 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.441 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:34:58.442 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:34:58.442 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.513 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:34:58.514 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:34:58.514 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.548 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:34:58.548 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:34:58.548 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.618 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:34:58.618 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:34:58.618 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.693 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:34:58.694 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:34:58.694 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.765 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:34:58.766 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:34:58.766 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:34:58.766 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.790 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:34:58.791 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:34:58.791 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.832 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:34:58.832 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:34:58.833 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:34:58.833 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.845 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:34:58.845 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:34:58.845 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.897 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:34:58.898 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:34:58.898 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:34:58.909 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:34:58.910 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:34:58.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:34:59.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:34:59.053 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:35:04.198 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:35:04.198 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:35:04.287 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:35:04.288 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:35:04.344 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:35:04.344 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:35:04.402 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:35:04.402 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:35:04.682 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:35:04.682 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:35:04.770 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:35:04.770 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:35:04.842 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:35:04.842 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:35:04.900 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:35:04.902 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/data/mon.a.tgz 2024-03-29T11:35:04.902 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-29T11:35:04.920 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.IS5XwSYWuv 2024-03-29T11:35:04.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IS5XwSYWuv 2024-03-29T11:35:05.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.IS5XwSYWuv 2024-03-29T11:35:05.110 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.IS5XwSYWuv is 126KB 2024-03-29T11:35:05.161 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.IS5XwSYWuv 2024-03-29T11:35:05.176 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/data/mon.c.tgz 2024-03-29T11:35:05.176 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-29T11:35:05.230 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.pONLWCHGUQ 2024-03-29T11:35:05.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.pONLWCHGUQ 2024-03-29T11:35:05.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.pONLWCHGUQ 2024-03-29T11:35:05.401 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.pONLWCHGUQ is 126KB 2024-03-29T11:35:05.452 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.pONLWCHGUQ 2024-03-29T11:35:05.466 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/data/mon.b.tgz 2024-03-29T11:35:05.466 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-29T11:35:05.484 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.T4ScGpcOXf 2024-03-29T11:35:05.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.T4ScGpcOXf 2024-03-29T11:35:05.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.T4ScGpcOXf 2024-03-29T11:35:05.674 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.T4ScGpcOXf is 126KB 2024-03-29T11:35:05.726 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.T4ScGpcOXf 2024-03-29T11:35:05.742 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:35:05.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:35:05.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:35:05.824 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:35:05.865 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:35:05.866 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:35:05.867 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/remote/smithi064/crash 2024-03-29T11:35:05.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:35:05.895 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/remote/smithi136/crash 2024-03-29T11:35:05.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:35:05.933 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:35:05.933 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:35:05.939 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:35:05.962 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2024-03-29T11:35:05.962 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40506.log 2024-03-29T11:35:06.004 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38852.log 2024-03-29T11:35:06.004 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2024-03-29T11:35:06.004 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2024-03-29T11:35:06.004 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38852.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:35:06.004 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40506.log.gz 2024-03-29T11:35:06.004 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41520.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38852.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40311.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41013.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38380.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38419.log 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40311.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39827.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38380.log.gz 2024-03-29T11:35:06.005 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39710.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39997.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41208.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38419.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39242.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39710.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39997.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39242.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41208.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38501.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39917.log 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39242.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38501.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-03-29T11:35:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34573.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38501.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34573.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39917.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40153.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34573.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40153.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39476.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-03-29T11:35:06.007 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39008.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39008.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40662.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40584.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39008.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40662.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40584.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40662.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39749.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40584.log.gz 2024-03-29T11:35:06.008 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38341.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39749.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40428.log: /var/log/ceph/ceph-client.admin.40389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40036.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-osd.1.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41559.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-03-29T11:35:06.009 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40075.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40701.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39164.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41559.log.gz 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39632.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40701.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40075.log.gz 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39632.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40701.log.gz 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40896.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39632.log.gz 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz --verbose 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr: -- /var/log/ceph/ceph-client.admin.38891.log 2024-03-29T11:35:06.010 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38774.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.3.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38774.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38891.log.gzgzip 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38302.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40857.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38774.log.gz 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40623.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38302.log: gzip -5 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40623.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38302.log.gz 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39203.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40623.log.gz 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38930.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38657.log: /var/log/ceph/ceph-client.admin.39203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39047.log 2024-03-29T11:35:06.011 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz/var/log/ceph/ceph-client.admin.38930.log: 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39203.log.gz 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40818.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39047.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38930.log.gz 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39047.log.gz 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40467.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: 85.4%/var/log/ceph/ceph-client.admin.41403.log: -- replaced with /var/log/ceph/ceph-client.admin.40818.log.gz 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38185.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz -5 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40467.log: /var/log/ceph/ceph-client.admin.38185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38618.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40467.log.gz 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38185.log.gz 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-03-29T11:35:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38263.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40779.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38969.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38263.log.gz 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39515.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38969.log.gz 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gzgzip 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40545.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39515.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz/var/log/ceph/ceph-client.admin.41598.log: 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40545.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39671.log 2024-03-29T11:35:06.013 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39515.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40974.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40545.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39866.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5/var/log/ceph/ceph-client.admin.39866.log: --verbose -- /var/log/ceph/ceph-client.admin.40192.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39671.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41286.log: gzip -5 86.2% --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39866.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40192.log.gz 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr: -- /var/log/ceph/ceph-client.admin.39788.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38462.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:35:06.014 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39788.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40350.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39788.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40350.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40350.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39086.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39359.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39437.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34864.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39437.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39086.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39437.log.gz 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38696.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40114.log 2024-03-29T11:35:06.015 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34864.log.gz 93.8% 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz/var/log/ceph/ceph-client.admin.40114.log: 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38696.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40114.log.gz 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38696.log.gz 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41130.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41442.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gzgzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28754.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 85.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41130.log: /var/log/ceph/ceph.tmp-client.admin.28754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38735.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28754.log.gz 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41130.log.gz 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39125.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41364.log 2024-03-29T11:35:06.016 INFO:teuthology.orchestra.run.smithi064.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39125.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38735.log.gz 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39125.log.gz 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41364.log: /var/log/ceph/ceph-client.admin.39281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40231.log 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.c.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41364.log.gz 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34967.log 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40231.log: /var/log/ceph/ceph-client.admin.34967.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40231.log.gz 2024-03-29T11:35:06.017 INFO:teuthology.orchestra.run.smithi064.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34967.log.gz 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34616.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.6.log: -- /var/log/ceph/ceph-client.admin.34496.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34576.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34656.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34616.log.gz 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34456.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34576.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34656.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34576.log.gz 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34536.log 2024-03-29T11:35:06.021 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34656.log.gz 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34456.log.gz 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34736.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34536.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 88.2% 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34696.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34536.log.gz 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34736.log.gz 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34696.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.7.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34696.log.gz 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:35:06.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:35:06.078 INFO:teuthology.orchestra.run.smithi136.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:35:17.427 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:35:17.883 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:35:18.060 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:35:18.097 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:35:21.807 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:35:21.836 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:35:22.246 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:35:22.286 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:35:58.684 INFO:teuthology.orchestra.run.smithi064.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:36:00.061 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:36:00.063 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T11:36:00.063 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m54.112s 2024-03-29T11:36:00.063 INFO:teuthology.orchestra.run.smithi064.stderr:user 2m38.011s 2024-03-29T11:36:00.063 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m5.282s 2024-03-29T11:36:14.040 INFO:teuthology.orchestra.run.smithi136.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:36:14.041 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T11:36:14.041 INFO:teuthology.orchestra.run.smithi136.stderr:real 1m8.051s 2024-03-29T11:36:14.041 INFO:teuthology.orchestra.run.smithi136.stderr:user 1m58.404s 2024-03-29T11:36:14.042 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m4.438s 2024-03-29T11:36:14.042 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:36:14.042 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/remote/smithi064/log 2024-03-29T11:36:14.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:36:37.778 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/remote/smithi136/log 2024-03-29T11:36:37.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:36:58.474 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:36:58.680 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c125ce5c954a486aa936bdfc170cd712 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:36:58.684 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:36:58.698 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:36:58.711 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:36:58.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:36:58.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:36:58.801 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:36:58.812 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:36:58.812 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:36:58.815 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:36:58.828 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:36:58.831 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:36:58.833 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:36:58.833 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-29T11:36:58.833 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +481us[ +473us] +/- 95ms 2024-03-29T11:36:58.833 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -80us[ -88us] +/- 58ms 2024-03-29T11:36:58.833 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -312us[ -312us] +/- 90ms 2024-03-29T11:36:58.833 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:58.836 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:36:58.836 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-29T11:36:58.836 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +481us[ +480us] +/- 95ms 2024-03-29T11:36:58.836 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -98us[ -100us] +/- 58ms 2024-03-29T11:36:58.836 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -296us[ -296us] +/- 90ms 2024-03-29T11:36:58.836 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:36:58.837 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:36:58.848 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:36:58.849 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:36:58.862 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:36:58.945 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:36:59.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:36:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:36:59.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:36:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:36:59.078 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:36:59.089 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:36:59.134 INFO:teuthology.task.internal:Duration was 1227.064586 seconds 2024-03-29T11:36:59.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:36:59.147 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:36:59.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:36:59.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:36:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:36:59.195 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:36:59.569 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:36:59.569 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T11:36:59.569 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:36:59.589 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T11:36:59.589 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:36:59.617 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:36:59.617 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:36:59.631 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:36:59.686 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:36:59.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:36:59.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:36:59.821 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:36:59.833 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:36:59.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:36:59.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:36:59.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:36:59.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:36:59.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:36:59.925 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-03-29T11:36:59.953 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-03-29T11:36:59.976 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:00.004 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:00.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:37:00.042 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:37:00.043 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/remote/smithi064 2024-03-29T11:37:00.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:37:00.080 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687/remote/smithi136 2024-03-29T11:37:00.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:37:00.115 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:37:00.115 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:37:00.117 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:37:00.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:37:00.176 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:37:00.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:37:00.191 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:37:00.191 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:37:00.193 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:37:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:37 /home/ubuntu/cephtest 2024-03-29T11:37:00.219 INFO:teuthology.orchestra.run.smithi136.stdout: 265618 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:37 /home/ubuntu/cephtest 2024-03-29T11:37:00.220 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:37:00.232 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:37:00.261 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:37:00.286 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-29T11:37:00.287 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:00.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.435895', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCxyo7B7VLJ3qOY7qa7kNNFZrBfrbkytLD55taMv4gDfkl4/Rx1bzjo7AJ1HANhDGdmag+SwnOr//NmJEn9Lwg4='} 2024-03-29T11:37:00.312 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-29T11:37:00.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:00.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.437411', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6b5dmWfdaAVZg90zCJVBO0VxQSE9VQJvV2te0DX5qtMVBzk4CufFZ1bdai4eSTz3kGaPT4rw7BrhuC4dddMUk='} 2024-03-29T11:37:00.354 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-29T11:37:00.355 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:37:00.378 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-29T11:37:00.379 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:00.396 INFO:teuthology.orchestra.console.smithi136:Power off 2024-03-29T11:37:00.396 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:37:00.405 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-29T11:37:00.418 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-03-29T11:37:00.418 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:00.442 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-29T11:37:04.406 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:04.444 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:04.518 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-29T11:37:04.554 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-29T11:37:08.523 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:08.555 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:08.635 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-29T11:37:08.665 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-03-29T11:37:08.737 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-29T11:37:08.767 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-03-29T11:37:08.871 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1227.06458568573 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c125ce5c954a486aa936bdfc170cd712 status: fail success: false 2024-03-29T11:37:08.872 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:37:08.954 INFO:teuthology.run:FAIL